【大数据系统架构师】1.2 大数据基础Hadoop 2.X

1. hadoop环境搭建

1.1 伪分布式环境搭建

1.1.1 伪分布式环境搭建

1.1.2 伪分布式搭建结果

hdfs可视化界面： http://od001:50070/dfshealth.html#tab-overview

yarn可视化界面： http://od001:8088/cluster

历史服务器可视化界面：http://od001:19888/

secondarynamenode可视化界面： http://od001:50090/status.html

1.1.3 批量启动脚本

#!/bin/bash
echo "启动namenode"
hadoop-daemon.sh start namenode
echo "启动datanode"
hadoop-daemon.sh start datanode
echo "启动resourcemanager"
yarn-daemon.sh start resourcemanager
echo "启动nodemanager"
yarn-daemon.sh start nodemanager
echo "启动historyserver"
mr-jobhistory-daemon.sh start historyserver
echo "启动secondarynamenode"
hadoop-daemon.sh start secondarynamenode

1.1.4 批量停止脚本

#!/bin/bash
echo "停止namenode"
hadoop-daemon.sh stop namenode
echo "停止datanode"
hadoop-daemon.sh stop datanode
echo "停止resourcemanager"
yarn-daemon.sh stop resourcemanager
echo "停止nodemanager"
yarn-daemon.sh stop nodemanager
echo "停止historyserver"
mr-jobhistory-daemon.sh stop historyserver
echo "停止secondarynamenode"
hadoop-daemon.sh stop secondarynamenode

 1.2 集群环境

1.2.1 集群环境搭建

1）克隆虚拟机

2） 使用root用户修改网卡信息

vi /etc/udev/rules.d/70-persistent-net.rules

vi /etc/sysconfig/network-scripts/ifcfg-eth0

1.2.2 资源规划

	od002	od003	od004
HDFS
	NameNode
	DataNode	DataNode	DataNode
			SecondaryNameNode
YARN		ResourceManger
	NodeManager	NodeManager	NodeManager
MapReduce
	JobHistoryServer

 

 1.3.3 配置

hfds

　　hadoop-env.sh

　　core-site.xml

　　hdfs-site.xml

　　slaves

yarn

　　yarn-env.sh

　　yarn-site.xml

　　slaves

mapreduce

　　mapred-env.sh

　　mapred-site.xml

1.3.4 配置同步

1）在od002、od003、od004分别使用ssh-keygen -t rsa命令，生成公私钥文件

2）在每台服务器的.ssh目录，使用

ssh-copy-id od002

ssh-copy-id od003

ssh-copy-id od004

配置ssh无密码登录

3）使用scp命令，同步配置文件

scp -r ./hadoop-2.5.0-cdh5.3.6/etc/ od003:/opt/modules/hadoop-2.5.0-cdh5.3.6

scp -r ./hadoop-2.5.0-cdh5.3.6/etc/ od004:/opt/modules/hadoop-2.5.0-cdh5.3.6

 1.3.5 启动命令

1）在od002上，执行start-dfs.sh命令

Starting namenodes on [od002]
od002: starting namenode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-namenode-od002.out
od004: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od004.out
od003: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od003.out
od002: starting datanode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-datanode-od002.out
Starting secondary namenodes [od004]
od004: starting secondarynamenode, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/hadoop-od-secondarynamenode-od004.out

2）在od003上，执行start-yarn.sh命令

starting yarn daemons
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-resourcemanager-od003.out
od004: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od004.out
od002: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od002.out
od003: starting nodemanager, logging to /opt/modules/hadoop-2.5.0-cdh5.3.6/logs/yarn-od-nodemanager-od003.out

3）验证环境

yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount input output001

1.3.6 基准测试

1）基本测试：服务启动、是否可用、简单的应用

hdfs读写操作

 

2）

1.3.9 集群环境搭建结果

hdfs可视化界面： http://od002:50070/dfshealth.html#tab-overview

yarn可视化界面： http://od003:8088/cluster

历史服务器可视化界面：http://od002:19888/

secondarynamenode可视化界面： http://od004:50090/status.html