hadoop2.2.0 单机开发搭建。 环境： 系统 centos 6.3 64位 jdk版本 oracle jdk 1.7 hadoop版本 2.2.0 使用linux用户 hadoop 目录配置 /home/hadoop 用户目录 /app/hadoop/hadoop-2.2.0 软件home /app/hadoop/dfs/name 数据和编辑文件 /app/hadoop/dfs/data
hadoop2.2.0 单机开发搭建。
环境：
| 系统 | centos 6.3 64位 |
| jdk版本 | oracle jdk 1.7 |
| hadoop版本 | 2.2.0 |
| 使用linux用户 | hadoop |
目录配置
| /home/hadoop | 用户目录 |
| /app/hadoop/hadoop-2.2.0 | 软件home |
| /app/hadoop/dfs/name | 数据和编辑文件 |
| /app/hadoop/dfs/data | 数据和编辑文件 |
| /app/hadoop/mapred/local | 存放数据 |
| /app/hadoop/mapred/system | 存放数据 |
1. 安装jdk
sudo vim /etc/profile
export java_home=/usr/lib/jvm/jdk1.7.0_45
export jre_home=$java_home/jre
export classpath=$java_home/lib:$classpath
export path=$java_home/bin:$path
source /etc/profile
2. ssh无密码登录
?
hadoop用户操作：
ssh-keygen -t dsa -p '' -f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub>> ~/.ssh/authorized_keys
root用户操作：
chmod go-w ?/home/hadoop/.ssh chmod 600 /home/hadoop/.ssh/authorized_keys
测试：
hadoop用户
[hadoop@hadoop01 ~]$ ssh localhost
3. 安装hadoop
可以自己下载源码包编译成适合本地native包
为了简单我直接下载编译好的hadoop包：
地址：http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.2.0/
解压到目录;
移动解压软件到软件目录:
/app/hadoop/hadoop-2.2.0
4. 修改hadoop参数文件
vim ?core-site.xml
fs.default.name
hdfs://hadoop-host:8020
the name of the defaultfile system. either the literal string "local" a host:port forndfs.
true
修改hdfs-site.xml
dfs.namenode.name.dir
file:/app/hadoop/dfs/name
true
dfs.datanode.data.dir
file:/app/hadoop/dfs/data
determineswhere on the local filesystem an dfs data node should store its blocks. if thisis a comma-delimited list of directories, then data will be stored in all nameddirectories, typically on different devices.directories that do not exist areignored.
true
dfs.replication
1
dfs.permissions
false
修改mapred-site.xml
mapreduce.framework.name
yarn
mapred.system.dir
file:/app/hadoop/mapred/system
true
mapred.local.dir
file:/app/hadoop/mapred/local
true
修改yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
如果要配置成集群环境则yarn-site.xml的配置如下：
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.shufflehandler
yarn.resourcemanager.address
master.hadoop:8032
yarn.resourcemanager.scheduler.address
master.hadoop:8030
yarn.resourcemanager.resource-tracker.address
master.hadoop:8031
yarn.resourcemanager.admin.address
master.hadoop:8033
yarn.resourcemanager.webapp.address
master.hadoop:8088
修改 hadoop-env.sh：
增加：
export java_home=/usr/java/jdk1.7.0_45
创建本地目录
mkdir –p ?/app/hadoop/dfs/name mkdir -p ?/app/hadoop/dfs/data mkdir –p ?/app/hadoop/mapred/local mkdir -p ?/app/hadoop/mapred/system启动hadoop
格式化namenode
[hadoop@hadoop01 ~]$ hdfs namenode –format
开启dfs守护进程
启动：start-all.sh
停止：stop-all.sh
开启yarn守护进程
启动：start-yarn.sh
停止：stop-yarn.sh
使用jps查看启动的进程：
[hadoop@ttpod sbin]$ jps
7621 namenode
11834 jps
7734 datanode
7881 secondarynamenode
10156 nodemanager
10053 resourcemanager
有以上内容说明已经启动
查看hadoop资源管理页面http://192.168.6.124:8088/
查看hdfs界面：http://192.168.6.124:50070
测试：
使用pi程序：
hadoop jar ?$hadoop_home/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 10 10
hadoop jar hadoop-mapreduce-examples-2.2.0.jar pi 10 10
| number of maps? = 10samples per map = 1013/12/13 16:27:23 warn util.nativecodeloader: unable to load native-hadoop library for your platform… using builtin-java classes where applicablewrote input for map #0
wrote input for map #1 wrote input for map #2 wrote input for map #3 wrote input for map #4 wrote input for map #5 wrote input for map #6 wrote input for map #7 wrote input for map #8 wrote input for map #9 starting job 13/12/13 16:27:24 info client.rmproxy: connecting to resourcemanager at /0.0.0.0:8032 13/12/13 16:27:25 info input.fileinputformat: total input paths to process : 10 13/12/13 16:27:25 info mapreduce.jobsubmitter: number of splits:10 13/12/13 16:27:25 info configuration.deprecation: user.name is deprecated. instead, use mapreduce.job.user.name 13/12/13 16:27:25 info configuration.deprecation: mapred.jar is deprecated. instead, use mapreduce.job.jar 13/12/13 16:27:25 info configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. instead, use mapreduce.map.speculative 13/12/13 16:27:25 info configuration.deprecation: mapred.reduce.tasks is deprecated. instead, use mapreduce.job.reduces 13/12/13 16:27:25 info configuration.deprecation: mapred.output.value.class is deprecated. instead, use mapreduce.job.output.value.class 13/12/13 16:27:25 info configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. instead, use mapreduce.reduce.speculative 13/12/13 16:27:25 info configuration.deprecation: mapreduce.map.class is deprecated. instead, use mapreduce.job.map.class 13/12/13 16:27:25 info configuration.deprecation: mapred.job.name is deprecated. instead, use mapreduce.job.name 13/12/13 16:27:25 info configuration.deprecation: mapreduce.reduce.class is deprecated. instead, use mapreduce.job.reduce.class 13/12/13 16:27:25 info configuration.deprecation: mapreduce.inputformat.class is deprecated. instead, use mapreduce.job.inputformat.class 13/12/13 16:27:25 info configuration.deprecation: mapred.input.dir is deprecated. instead, use mapreduce.input.fileinputformat.inputdir 13/12/13 16:27:25 info configuration.deprecation: mapred.output.dir is deprecated. instead, use mapreduce.output.fileoutputformat.outputdir 13/12/13 16:27:25 info configuration.deprecation: mapreduce.outputformat.class is deprecated. instead, use mapreduce.job.outputformat.class 13/12/13 16:27:25 info configuration.deprecation: mapred.map.tasks is deprecated. instead, use mapreduce.job.maps 13/12/13 16:27:25 info configuration.deprecation: mapred.output.key.class is deprecated. instead, use mapreduce.job.output.key.class 13/12/13 16:27:25 info configuration.deprecation: mapred.working.dir is deprecated. instead, use mapreduce.job.working.dir 13/12/13 16:27:25 info mapreduce.jobsubmitter: submitting tokens for job: job_1386923206015_0001 13/12/13 16:27:26 info impl.yarnclientimpl: submitted application application_1386923206015_0001 to resourcemanager at /0.0.0.0:8032 13/12/13 16:27:26 info mapreduce.job: the url to track the job: http://ttpod:8088/proxy/application_1386923206015_0001/ 13/12/13 16:27:26 info mapreduce.job: running job: job_1386923206015_0001 13/12/13 16:27:34 info mapreduce.job: job job_1386923206015_0001 running in uber mode : false 13/12/13 16:27:34 info mapreduce.job:? map 0% reduce 0% 13/12/13 16:27:56 info mapreduce.job:? map 60% reduce 0% 13/12/13 16:28:13 info mapreduce.job:? map 100% reduce 0% 13/12/13 16:28:14 info mapreduce.job:? map 100% reduce 100% 13/12/13 16:28:15 info mapreduce.job: job job_1386923206015_0001 completed successfully 13/12/13 16:28:16 info mapreduce.job: counters: 43 file system counters file: number of bytes read=226 file: number of bytes written=871752 file: number of read operations=0 file: number of large read operations=0 file: number of write operations=0 hdfs: number of bytes read=2610 hdfs: number of bytes written=215 hdfs: number of read operations=43 hdfs: number of large read operations=0 hdfs: number of write operations=3 job counters launched map tasks=10 launched reduce tasks=1 data-local map tasks=10 total time spent by all maps in occupied slots (ms)=185391 total time spent by all reduces in occupied slots (ms)=15412 map-reduce framework map input records=10 map output records=20 map output bytes=180 map output materialized bytes=280 input split bytes=1430 combine input records=0 combine output records=0 reduce input groups=2 reduce shuffle bytes=280 reduce input records=20 reduce output records=0 spilled records=40 shuffled maps =10 failed shuffles=0 merged map outputs=10 gc time elapsed (ms)=1841 cpu time spent (ms)=7600 physical memory (bytes) snapshot=2507419648 virtual memory (bytes) snapshot=9588948992 total committed heap usage (bytes)=1944584192 shuffle errors bad_id=0 connection=0 io_error=0 wrong_length=0 wrong_map=0 wrong_reduce=0 file input format counters bytes read=1180 file output format counters bytes written=97 job finished in 51.367 seconds estimated value of pi is 3.20000000000000000000 |
正常.
如出现什么异常请自行去查看运行日志。
【说明】：本文章由站长整理发布，文章内容不代表本站观点，如文中有侵权行为，请与本站客服联系(QQ:)！