java是hadoop开发的标准官方语言，本文下载了官方的wordcount.java并对其进行了编译和打包，然后使用测试数据运行了该hadoop程序。 这里假定已经装好了hadoop的环境，在linux下运行hadoop命令能够正常执行； 下载java版本的wordcount.java程序。 将wordcount
java是hadoop开发的标准官方语言，本文下载了官方的wordcount.java并对其进行了编译和打包，然后使用测试数据运行了该hadoop程序。
这里假定已经装好了hadoop的环境，在linux下运行hadoop命令能够正常执行；
下载java版本的wordcount.java程序。
将wordcount.java复制到linux下的一个目录，这里我复制到/home/crazyant/hadoop_wordcount[crazyant@dev.mechine hadoop_wordcount]$ ll
total 4
-rwxr–r–? 1 crazyant crazyant 1921 aug 16 20:03 wordcount.java
在该目录（/home/crazyant/hadoop_wordcount）下创建wordcount_classes目录，用于存放编译wordcount.java生成的class文件。[crazyant@dev.mechine hadoop_wordcount]$ mkdir wordcount_classes
[crazyant@dev.mechine hadoop_wordcount]$ ll
total 8
drwxrwxr-x? 2 crazyant crazyant 4096 aug 16 20:07 wordcount_classes
-rwxr–r–? 1 crazyant crazyant 1921 aug 16 20:03 wordcount.java
编译wordcount.java文件，其中-classpath选项表示要引用hadoop官方的包，-d选项表示要将编译后的class文件生成的目标目录。[crazyant@dev.mechine hadoop_wordcount]$ javac -classpath /home/crazyant/app/hadoop/hadoop-2-core.jar -d wordcount_classes wordcount.java
[crazyant@dev.mechine hadoop_wordcount]$ ll -r
.:
total 8
drwxrwxr-x? 3 crazyant crazyant 4096 aug 16 20:09 wordcount_classes
-rwxr–r–? 1 crazyant crazyant 1921 aug 16 20:03 wordcount.java
./wordcount_classes:
total 4
drwxrwxr-x? 3 crazyant crazyant 4096 aug 16 20:09 org
./wordcount_classes/org:
total 4
drwxrwxr-x? 2 crazyant crazyant 4096 aug 16 20:09 myorg
./wordcount_classes/org/myorg:
total 12
-rw-rw-r–? 1 crazyant crazyant 1546 aug 16 20:09 wordcount.class
-rw-rw-r–? 1 crazyant crazyant 1938 aug 16 20:09 wordcount$map.class
-rw-rw-r–? 1 crazyant crazyant 1611 aug 16 20:09 wordcount$reduce.class
然后将编译后的class文件打包：[crazyant@dev.mechine hadoop_wordcount]$ jar -cvf wordcount.jar -c wordcount_classes/ .
added manifest
adding: org/(in = 0) (out= 0)(stored 0%)
adding: org/myorg/(in = 0) (out= 0)(stored 0%)
adding: org/myorg/wordcount$map.class(in = 1938) (out= 798)(deflated 58%)
adding: org/myorg/wordcount$reduce.class(in = 1611) (out= 649)(deflated 59%)
adding: org/myorg/wordcount.class(in = 1546) (out= 749)(deflated 51%)
[crazyant@dev.mechine hadoop_wordcount]$ ll
total 12
drwxrwxr-x? 3 crazyant crazyant 4096 aug 16 20:09 wordcount_classes
-rw-rw-r–? 1 crazyant crazyant 3169 aug 16 20:11 wordcount.jar
-rwxr–r–? 1 crazyant crazyant 1921 aug 16 20:03 wordcount.java
在本地用echo生成一个文件，用于输入数据：[crazyant@dev.mechine hadoop_wordcount]$ echo “hello world, hello crazyant, i am the ant, i am your brother” > inputfile
[crazyant@dev.mechine hadoop_wordcount]$ more inputfile
hello world, hello crazyant, i am the ant, i am your brother
在hadoop上建立一个目录，里面建立输入文件的目录[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -mkdir /app/word_count/input
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count
found 1 items
drwxr-xr-x?? 3 czt czt????????? 0 2013-08-16 20:16 /app/word_count/input
将本地刚刚写的的inputfile上传到hadoop上的input目录[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -put inputfile /app/word_count/input
[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/input
found 1 items
-rw-r–r–?? 3 czt czt???????? 61 2013-08-16 20:18 /app/word_count/input/inputfile
运行jar，以建立的input目录作为输入参数[crazyant@dev.mechine hadoop_wordcount]$ hadoop jar wordcount.jar org.myorg.wordcount /app/word_count/input /app/word_count/output
13/08/16 20:19:38 warn mapred.jobclient: use genericoptionsparser for parsing the arguments. applications should implement tool for the same.
13/08/16 20:19:40 info util.nativecodeloader: loaded the native-hadoop library
13/08/16 20:19:40 info compress.lzocodec: successfully loaded & initialized native-lzo library
13/08/16 20:19:40 info compress.lzmacodec: successfully loaded & initialized native-lzma library
13/08/16 20:19:40 info compress.quicklzcodec: successfully loaded & initialized native-quicklz library
13/08/16 20:19:40 info mapred.fileinputformat: total input paths to process : 1
13/08/16 20:19:41 info mapred.jobclient: splits size : 61
13/08/16 20:19:41 info mapred.jobclient: running job: job_20130813122541_105844
13/08/16 20:19:43 info mapred.jobclient:? map 0% reduce 0%
13/08/16 20:19:57 info mapred.jobclient:? map 24% reduce 0%
13/08/16 20:20:07 info mapred.jobclient:? map 93% reduce 0%
13/08/16 20:20:16 info mapred.jobclient:? map 100% reduce 1%
13/08/16 20:20:26 info mapred.jobclient:? map 100% reduce 61%
13/08/16 20:20:36 info mapred.jobclient:? map 100% reduce 89%
13/08/16 20:20:47 info mapred.jobclient:? map 100% reduce 96%
13/08/16 20:20:57 info mapred.jobclient:? map 100% reduce 98%
13/08/16 20:21:00 info mapred.jobclient: updating completed job! ignoring …
13/08/16 20:21:00 info mapred.jobclient: updating completed job! ignoring …
13/08/16 20:21:00 info mapred.jobclient: job complete: job_20130813122541_105844
13/08/16 20:21:00 info mapred.jobclient: counters: 19
13/08/16 20:21:00 info mapred.jobclient:?? file systems
13/08/16 20:21:00 info mapred.jobclient:???? hdfs bytes read=1951
13/08/16 20:21:00 info mapred.jobclient:???? hdfs bytes written=68
13/08/16 20:21:00 info mapred.jobclient:???? local bytes read=5174715
13/08/16 20:21:00 info mapred.jobclient:???? local bytes written=256814
13/08/16 20:21:00 info mapred.jobclient:?? job counters
13/08/16 20:21:00 info mapred.jobclient:???? launched reduce tasks=100
13/08/16 20:21:00 info mapred.jobclient:???? rack-local map tasks=61
13/08/16 20:21:00 info mapred.jobclient:???? original_reduces=100
13/08/16 20:21:00 info mapred.jobclient:???? launched map tasks=61
13/08/16 20:21:00 info mapred.jobclient:???? miss_scheduled_reduces=15
13/08/16 20:21:00 info mapred.jobclient:?? task_statistics
13/08/16 20:21:00 info mapred.jobclient:???? total map slot time=34
13/08/16 20:21:00 info mapred.jobclient:???? attempt_0 map task count=61
13/08/16 20:21:00 info mapred.jobclient:???? total reduce slot time=892
13/08/16 20:21:00 info mapred.jobclient:?? map-reduce framework
13/08/16 20:21:00 info mapred.jobclient:???? reduce input groups=9
13/08/16 20:21:00 info mapred.jobclient:???? combine output records=0
13/08/16 20:21:00 info mapred.jobclient:???? map input records=1
13/08/16 20:21:00 info mapred.jobclient:???? reduce output records=9
13/08/16 20:21:00 info mapred.jobclient:???? map input bytes=61
13/08/16 20:21:00 info mapred.jobclient:???? combine input records=0
13/08/16 20:21:00 info mapred.jobclient:???? reduce input records=9
查看output目录是否有结果[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/output??????????????????????????????????????????????????? found 100 items
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00000
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00001
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00002
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00003
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00004
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00005
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00006
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00007
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00008
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00009
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00010
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00011
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00012
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00013
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00014
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00015
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00016
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00017
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00018
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00019
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00020
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00021
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00022
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00023
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00024
-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00025
将该目录下所有文本文件合并后下载到本地[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -getmerge /app/word_count/output wordcount_result
[crazyant@dev.mechine hadoop_wordcount]$ ls
inputfile? wordcount_classes? wordcount.jar? wordcount.java? wordcount_result
查看一下下载下来的计算结果[crazyant@dev.mechine hadoop_wordcount]$ more wordcount_result
i?????? 2
your??? 1
crazyant,?????? 1
brother 1
hello?? 2
am????? 2
world,? 1
the???? 1
ant,??? 1
统计结果正确；
参考文章：http://hadoop.apache.org/docs/r0.18.3/mapred_tutorial.html#example%3a+wordcount+v1.0
【说明】：本文章由站长整理发布，文章内容不代表本站观点，如文中有侵权行为，请与本站客服联系(QQ:)！