详细版在虚拟机安装和使用hadoop分布式集群
集群模式:
一台master 192.168.85.2
一台slave 192.168.85.3
jdk jdk1.8.0_74(版本不重要,看喜欢)
hadoop版本 2.7.2(版本不重要,2.*都差不多)
本文从安装Ubuntu14.04后开始一步步搭建hadoop集群:
简单说一下虚拟机linux系统的搭建:
因为要搭建hadoop集群,所以预计至少两台虚拟机,这个不着急,我们可以布置一台然后克隆出另一台,然后稍微改动一下配置
我用的镜像是ubuntu-14.04.3-server-amd64.iso,为了主机连接和网络连接建立两个网卡,相关内容可以查看另外一篇博文:本机上搭建虚拟机的网络玩法,安装过程中注意安装openssh服务就好了,安装好之后可以用工具ssh到虚拟机上面操作更方便.安装的时候可以直接指定主机名为master比较好识别,用户名指定为hadoop
1.安装jdk
查看是否安装jdk
java -version
如果未安装参考:Ubuntu系统如何卸载并安装新版本的jdk(permission denied问题),已安装则跳过此步
2.下载hadoop
我下载的地址http://mirror.bit.edu.cn/apache/hadoop/common/,上面会有很多种版本可以选择,对试用来说都是一样的.随便下一个
通过ftp或者ssh传送到虚拟机上解压:
tar zxvf hadoop-2.7.2.tar.gz
重命名:
mv hadoop-2.7.2 hadoop
查看安装目录:
hadoop@master:~/hadoop$ pwd
/home/hadoop/hadoop
接下来配置多个配置文件,配置文件集中在安装目录下的的etc/hadoop下,我们将目录切换到该目录下方便操作,我将配置的内容贴出来:
slaves文件
vi slaves
内容改为
master
core-site.xml文件
vi core-site.xml
在<configuration>标签中添加如下内容:
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
hdfs-site.xml
vi hdfs-site.xml
添加:
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hadoop/dfs/name</value>
<description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hadoop/dfs/data</value>
<description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
mapred-site.xml,这个文件需要从模板中复制一份过来:
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
添加
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn-site.xml
vi yarn-site.xml
添加:
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
到这边理论上hadoop已经可以跑了.但是在hadoop启动过程中因为脚本的限定可能会报一些环境配置错误,我经过实践为了一绝后患先将该配置的东西配置好
首先是的java_home配置:
vi hadoop-env.sh
修改
export JAVA_HOME=/home/hadoop/jdk1.8.0_74
然后添加hadoop环境变量配置(在java配置下面添加就行):
export HADOOP_DEV_HOME=/home/hadoop/hadoop export PATH=$PATH:$HADOOP_DEV_HOME/bin export PATH=$PATH:$HADOOP_DEV_HOME/sbin export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME} export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME} export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME} export YARN_HOME=${HADOOP_DEV_HOME} export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
保存更改
source ~/.bashrc
格式化hdfs
bin/hdfs namenode -format
在克隆虚拟机之前先将主机配置好.
vi /etc/hosts
修改
127.0.0.1 localhost #127.0.1.1 master 192.168.85.2 master 192.168.85.3 slave1
克隆虚拟机,并启动克隆的机器.修改主机名和ip
vi /etc/hostname
修改为slave1
vi /etc/network/interfaces
看到
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet static address 192.168.85.2 netmask 255.255.255.0
修改为
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet static address 192.168.85.3 netmask 255.255.255.0
重启机器
到此,hadoop集群就搭建完了.
安装两台机器后,需要让master无密码登录到slave上面
ssh localhost
cd ~/.ssh
ssh-keygen -t rsa
一直确认即可;
Master 节点需能无密码 ssh 本机，这一步还是在 Master 节点上执行：
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
完成后可以使用 ssh Master 验证一下。接着将公匙传输到 Slave1 节点：
scp ~/.ssh/id_rsa.pub hadoop@Slave1:/home/hadoop/
scp时会要求输入Slave1上hadoop用户的密码(hadoop)，输入完成后会提示传输完毕。
接着在 Slave1节点 上将ssh公匙保存到相应位置，执行
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
我们来测试一下是否可以运行:
sbin/start-dfs.sh sbin/start-yarn.sh
这个命令启动了master和slave上面的东西,用jps查看内容
hadoop@master:~$ jps 1632 SecondaryNameNode 4581 Jps 1782 ResourceManager 1402 NameNode
在slave1中执行jps
4586 Jps 3210 DataNode 3356 NodeManager
登录http://192.168.85.2:50070/可以看到master和slave的分布以及启动状况
执行经典案例wordcount.
新建一个text1.txt并上传到集群
cd mkdir input cd input echo "hello world" > test1.txt hadoop fs –mkdir input
最后一条命令可能会报错,报错找不到input文件夹,那是因为hdfs初始化还没有根目录,加上/就好了
hadoop fs –mkdir /input
查看文件:
hadoop@master:~/hadoop$ hadoop fs -ls / Found 1 items drwxr-xr-x - hadoop supergroup 0 2016-05-10 10:36 /input
上传文件到input中并查看
hadoop@master:~/hadoop$ hadoop fs -put ../input/*.txt /input
hadoop@master:~/hadoop$ hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2016-05-10 10:38 /input
hadoop@master:~/hadoop$ hadoop fs -ls /input
Found 2 items
-rw-r--r-- 1 hadoop supergroup 12 2016-05-10 10:38 /input/test1.txt
-rw-r--r-- 1 hadoop supergroup 13 2016-05-10 10:38 /input/test2.txt
接下来就是用hadoop自带的一个脚本运行该文件,计算单词数
hadoop/bin/hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input/test1.txt output2
格式是hadoop脚本+jar命令+jar脚本+方法+输入文件+输出文件.
job开始执行输出
16/05/10 10:44:14 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.85.2:8032 16/05/10 10:44:15 INFO input.FileInputFormat: Total input paths to process : 1 16/05/10 10:44:15 INFO mapreduce.JobSubmitter: number of splits:1 16/05/10 10:44:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462879083278_0001 16/05/10 10:44:16 INFO impl.YarnClientImpl: Submitted application application_1462879083278_0001 16/05/10 10:44:16 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1462879083278_0001/ 16/05/10 10:44:16 INFO mapreduce.Job: Running job: job_1462879083278_0001 16/05/10 10:44:30 INFO mapreduce.Job: Job job_1462879083278_0001 running in uber mode : false 16/05/10 10:44:30 INFO mapreduce.Job: map 0% reduce 0% 16/05/10 10:44:40 INFO mapreduce.Job: map 100% reduce 0% 16/05/10 10:44:47 INFO mapreduce.Job: map 100% reduce 100% 16/05/10 10:44:47 INFO mapreduce.Job: Job job_1462879083278_0001 completed successfully 16/05/10 10:44:47 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=30 FILE: Number of bytes written=234875 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=111 HDFS: Number of bytes written=16 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=7470 Total time spent by all reduces in occupied slots (ms)=4602 Total time spent by all map tasks (ms)=7470 Total time spent by all reduce tasks (ms)=4602 Total vcore-milliseconds taken by all map tasks=7470 Total vcore-milliseconds taken by all reduce tasks=4602 Total megabyte-milliseconds taken by all map tasks=7649280 Total megabyte-milliseconds taken by all reduce tasks=4712448 Map-Reduce Framework Map input records=1 Map output records=2 Map output bytes=20 Map output materialized bytes=30 Input split bytes=99 Combine input records=2 Combine output records=2 Reduce input groups=2 Reduce shuffle bytes=30 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=229 CPU time spent (ms)=2730 Physical memory (bytes) snapshot=298352640 Virtual memory (bytes) snapshot=3748110336 Total committed heap usage (bytes)=139145216 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=12 File Output Format Counters Bytes Written=16
http://master:8088/proxy/application_1462879083278_0001/可以查看当前job的运行状态,在运行过程中可以查看.看到map 100% reduce 100%就是运行成功了,可以登录http://192.168.85.2:8088/cluster查看具体信息
最后关闭hadoop集群
sbin/stop-dfs.sh sbin/stop-yarn.sh
是不是很简单呢.
