【大数据系统架构师】1.2 大数据基础Hadoop 2.X
1. hadoop环境搭建
1.1 伪分布式环境搭建
1.1.1 伪分布式环境搭建
1.1.2 伪分布式搭建结果
hdfs可视化界面: http://od001:50070/dfshealth.html#tab-overview
yarn可视化界面: http://od001:8088/cluster
历史服务器可视化界面:http://od001:19888/
secondarynamenode可视化界面: http://od001:50090/status.html
1.1.3 批量启动脚本
#!/bin/bash echo "启动namenode" hadoop-daemon.sh start namenode echo "启动datanode" hadoop-daemon.sh start datanode echo "启动resourcemanager" yarn-daemon.sh start resourcemanager echo "启动nodemanager" yarn-daemon.sh start nodemanager echo "启动historyserver" mr-jobhistory-daemon.sh start historyserver echo "启动secondarynamenode" hadoop-daemon.sh start secondarynamenode
1.1.4 批量停止脚本
#!/bin/bash echo "停止namenode" hadoop-daemon.sh stop namenode echo "停止datanode" hadoop-daemon.sh stop datanode echo "停止resourcemanager" yarn-daemon.sh stop resourcemanager echo "停止nodemanager" yarn-daemon.sh stop nodemanager echo "停止historyserver" mr-jobhistory-daemon.sh stop historyserver echo "停止secondarynamenode" hadoop-daemon.sh stop secondarynamenode
1.2 集群环境
1.2.1 集群环境搭建
1)克隆虚拟机
2) 使用root用户修改网卡信息
vi /etc/udev/rules.d/70-persistent-net.rules
vi /etc/sysconfig/network-scripts/ifcfg-eth0
1.2.2 资源规划
od002 | od003 | od004 | |
HDFS | |||
NameNode | |||
DataNode | DataNode | DataNode | |
SecondaryNameNode | |||
YARN | ResourceManger | ||
NodeManager | NodeManager | NodeManager | |
MapReduce | |||
JobHistoryServer | |||
1.3.3 配置
hfds
hadoop-env.sh
core-site.xml
hdfs-site.xml
slaves
yarn
yarn-env.sh
yarn-site.xml
slaves
mapreduce
mapred-env.sh
mapred-site.xml
1.3.4 配置同步
1)在od002、od003、od004分别使用ssh-keygen -t rsa命令,生成公私钥文件
2)在每台服务器的.ssh目录,使用
ssh-copy-id od002
ssh-copy-id od003
ssh-copy-id od004
配置ssh无密码登录
3)使用scp命令,同步配置文件
scp -r ./hadoop-2.5.0-cdh5.3.6/etc/ od003:/opt/modules/hadoop-2.5.0-cdh5.3.6 scp -r ./hadoop-2.5.0-cdh5.3.6/etc/ od004:/opt/modules/hadoop-2.5.0-cdh5.3.6
1.3.5 启动命令
1)在od002上,执行start-dfs.sh命令
Starting namenodes on [od002] od002: starting namenode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-namenode-od002.out od004: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od004.out od003: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od003.out od002: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od002.out Starting secondary namenodes [od004] od004: starting secondarynamenode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-secondarynamenode-od004.out
2)在od003上,执行start-yarn.sh命令
starting yarn daemons starting resourcemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-resourcemanager-od003.out od004: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od004.out od002: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od002.out od003: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od003.out
3)验证环境
yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount input output001
1.3.6 基准测试
1)基本测试:服务启动、是否可用、简单的应用
hdfs读写操作
2)
1.3.9 集群环境搭建结果
hdfs可视化界面: http://od002:50070/dfshealth.html#tab-overview
yarn可视化界面: http://od003:8088/cluster
历史服务器可视化界面:http://od002:19888/
secondarynamenode可视化界面: http://od004:50090/status.html
![]() |
作者:沙漏哟 出处:计算机的未来在于连接 本文版权归作者和博客园共有,欢迎转载,请留下原文链接 微信随缘扩列,聊创业聊产品,偶尔搞搞技术 |