3、hadoop的分布式安装
1.基本信息
版本 | 2.7.3 |
---|---|
安装机器 | 三台机器 |
账号 | hadoop |
源路径 | /opt/software/hadoop-2.7.3.tar.gz |
目标路径 | /opt/hadoop -> /opt/hadoop-2.7.3 |
依赖关系 | 无 |
2.安装过程
1).切换到hadoop账户,通过tar -zxvf命令将hadoop解压缩至目的安装目录:
[root @bgs -5p173-wangwenting opt]# su hadoop [hadoop @bgs -5p173-wangwenting opt]$ cd /opt/software [hadoop @bgs -5p173-wangwenting software]$ tar -zxvf hadoop-${version}.tar.gz -C /opt [hadoop @bgs -5p173-wangwenting software]$ cd /opt [hadoop @bgs -5p173-wangwenting opt]$ ln -s /opt/hadoop-${version} /opt/hadoop |
2).创建tmpdir目录:
[hadoop @bgs -5p173-wangwenting opt]$ cd /opt/hadoop [hadoop @bgs -5p173-wangwenting hadoop]$ mkdir -p tmpdir |
3).配置hadoop-env.sh文件:
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ mkdir -p /opt/hadoop/pids [hadoop @bgs -5p173-wangwenting hadoop]$ vi hadoop-env.sh 在hadoop-env.sh文件中添加如下配置: export JAVA_HOME=/opt/java export HADOOP_PID_DIR=/opt/hadoop/pids |
4.配置mapred-env.sh文件:
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ vim mapred-env.sh 在mapred-env.sh文件中添加如下配置: export JAVA_HOME=/opt/java |
5.配置core-site.xml文件
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ vim core-site.xml 在core-site.xml文件中添加如下配置: <configuration> <property> //namenode的临时工作目录 <name>hadoop.tmp.dir</name> <value>/opt/hadoop/tmpdir</value> </property> <property> //hdfs的入口,告诉namenode在那个机器上,端口号是什么。 <name>fs.defaultFS</name> <value>hdfs: //bgs-5p173-wangwenting:9000</value> </property> <property> <name>io.file.buffer.size</name> <value> 131072 </value> </property> <property> <name>fs.trash.interval</name> <value> 1440 </value> </property> </configuration> |
6.配置hdfs-site.xml文件
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ vim hdfs-site.xml 在hdfs-site.xml文件中添加如下配置: <configuration> <property> //副本数量,一般是小于等于datanode的数量, <name>dfs.replication</name> <value> 2 </value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/opt/hadoop/tmpdir/dfs/name</value> </property> <property> <name>dfs.datanode.name.dir</name> <value>file:/opt/hadoop/tmpdir/dfs/data</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value> true </value> </property> <property> <name>dfs.secondary.http.address</name> <value>bgs-5p173-wangwenting: 50090 </value> </property> </configuration> |
7.配置mapred-site.xml文件
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ vi mapred-site.xml 在mapred-site.xml文件中添加如下配置: <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>bgs-5p173-wangwenting: 10020 </value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>bgs-5p173-wangwenting: 19888 </value> </property> </configuration> |
8.配置yarn-site.xml文件:
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/ [hadoop @bgs -5p173-wangwenting hadoop]$ vim yarn-site.xml 在yarn-site.xml文件中添加如下配置: <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle. class </name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>bgs-5p173-wangwenting: 8030 </value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>bgs-5p173-wangwenting: 8031 </value> </property> <property> <name>yarn.resourcemanager.address</name> <value>bgs-5p173-wangwenting: 8032 </value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>bgs-5p173-wangwenting: 8033 </value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>bgs-5p173-wangwenting: 8088 </value> </property> <!-- Site specific YARN configuration properties --> </configuration> |
9.配置hadoop运行的环境变量
[hadoop @bgs -5p173-wangwenting hadoop]$ vim /etc/profile export HADOOP_HOME=/opt/hadoop export PATH=$HADOOP_HOME/bin:$PATH 配置成功后,执行source /etc/profile使配置生效 [hadoop @bgs -5p173-wangwenting hadoop]$ source /etc/profile |
10.修改slaves文件:
[hadoop @bgs -5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop [hadoop @bgs -5p173-wangwenting hadoop]$ vim slaves 在slaves文件中添加 //datanode的节点的位置 bgs-5p173-wangwenting bgs-5p174-wangwenting bgs-5p175-wangwenting |
11.在bgs-5p173-wangwenting上复制hadoop-2.7.3到hadoop@bgs-5p174-wangwenting和hadoop@bgs-5p174-wangwenting机器并按照步骤9修改环境变量并执行以下操作:
[hadoop @bgs -5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop @bgs -5p174-wangwenting:/opt/ [hadoop @bgs -5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop [hadoop @bgs -5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop @bgs -5p175-wangwenting:/opt/ [hadoop @bgs -5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop |
12.格式化namenode,启动hadoop,并启动jobhistory服务:
[hadoop @bgs -5p173-wangwenting hadoop]$ hadoop namenode -format [hadoop @bgs -5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/start-all.sh [hadoop @bgs -5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/mr-jobhistory-daemon.sh start historyserver start-all.sh包含dfs和yarn两个模块的启动,分别为start-dfs.sh 、 start-yarn.sh,所以dfs和yarn可以单独启动。 注意:如果datanode没有启动起来,看看是不是tmpdir中有之前的脏数据,删除这个目录其他两台机器也要删除。 |
13.检查每台机器的服务,hadoop@bgs-5p173-wangwenting、hadoop@bgs-5p174-wangwenting、hadoop@bgs-5p175-wangwenting三台机器上分别输入jps:
[hadoop @bgs -5p173-wangwenting ~]$ jps 24429 Jps 22898 ResourceManager 23229 NodeManager 24383 JobHistoryServer 22722 SecondaryNameNode 22488 NameNode 6945 DataNode [ahdoop @bgs -5p174-wangwenting ~]$ jps 7650 DataNode 7788 NodeManager 8018 Jps [hadoop @bgs -5p175-wangwenting ~]$ jps 28407 Jps 28038 DataNode 28178 NodeManager 如果三台机器正常输出上述内容,则表示hadoop集群的服务正常工作。 |
访问hadoop的服务页面:在浏览器中输入如下地址
http://bgs-5p173-wangwenting:8088
http://bgs-5p173-wangwenting:50070
http://bgs-5p173-wangwenting:19888
如能正常反应,则表明上述三类服务正常运行.