hadopp 环境搭建
前序:
首先准备三个虚拟机节点。
配置hosts文件:每个节点都 如下配置:
vi /etc/hosts
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/2af7f30e-6953-430d-b0da-90f2ea72aff2.png)
1、 每个结点分别产生公私密钥
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/2faea710-40b8-4f7a-bc93-432446e31876.png)
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/9b04fbbc-31d1-4566-962c-4df1f7376f73.png)
以上命令是产生公私密钥,产生目录在用户主目录下的.ssh目录中。
Id_dsa.pub为公钥,id_dsa为私钥,紧接着将公钥文件复制成authorized_keys文件,这个步骤是必须的,过程如下
[root@master .ssh]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
单机回环ssh免密码登录测试
如果ssh localhost 返回 ![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/534de274-730c-4613-a93d-353ef92585d8.png)
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/534de274-730c-4613-a93d-353ef92585d8.png)
则用yum下载SSH :
yum -y install openssh-clients
以下信息表示操作成功,单点回环SSH登录及注销成功,这将为后续跨子结点SSH远程免密码登录作好准备。
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/e17824d3-2cc4-4d04-89da-ac1d66f65bf7.png)
2、其他节点也是如此操作即可。
3、让主结点(master)能通过SSH免密码登录两个子结点(slave)
为了实现这个功能,两个slave结点的公钥文件中必须要包含主结点的公钥信息,这样
当master就可以顺利安全地访问这两个slave结点了。操作过程如下:
[root@node1 ~]# scp root@master:~/.ssh/id_dsa.pub ./master_dsa.pub
The authenticity of host 'master (30.96.76.220)' can't be established.
RSA key fingerprint is ae:8c:7f:00:df:40:b8:ec:20:4b:53:78:98:46:8a:c5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,30.96.76.220' (RSA) to the list of known hosts.
root@master's password:
id_dsa.pub 100% 601 0.6KB/s 00:00
[root@node1 .ssh]# cat master_dsa.pub >> authorized_keys
如上过程显示了node1结点通过scp命令远程登录master结点,并复制master的公钥文件到当前的目录下,这一过程需要密码验证。接着,将master结点的公钥文件追加至authorized_keys文件中,通过这步操作,如果不出问题,master结点就可以通过ssh远程免密码连接node1结点了。在master结点中操作如下:
[root@master .ssh]# ssh node1
The authenticity of host 'node1 (30.96.76.221)' can't be established.
RSA key fingerprint is ae:8c:7f:00:df:40:b8:ec:20:4b:53:78:98:46:8a:c5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,30.96.76.221' (RSA) to the list of known hosts.
root@node1's password:
Last login: Fri Sep 9 17:42:15 2016 from localhost
[root@node1 ~]# ll
总用量 24
-rw-------. 1 root root 1144 9月 9 18:11 anaconda-ks.cfg
-rw-r--r--. 1 root root 13231 9月 9 18:10 install.log
-rw-r--r--. 1 root root 3482 9月 9 18:09 install.log.syslog
[root@node1 ~]# pwd
/root
[root@node1 ~]# exit
logout
Connection to node1 closed.
[root@master .ssh]#
表面上看,这两个结点的ssh免密码登录已经配置成功,但是我们还需要对主结点master也要进行上面的同样工作,这一步有点让人困惑,但是这是有原因的,具体原因现在也说不太好,据说是真实物理结点时需要做这项工作,因为jobtracker有可能会分布在其它结点上,jobtracker有不存在master结点上的可能性。
[root@master .ssh]# scp root@master:~/.ssh/id_dsa.pub ./master_dsa.pub
The authenticity of host 'master (30.96.76.220)' can't be established.
RSA key fingerprint is ae:8c:7f:00:df:40:b8:ec:20:4b:53:78:98:46:8a:c5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,30.96.76.220' (RSA) to the list of known hosts.
id_dsa.pub 100% 601 0.6KB/s 00:00
[root@master .ssh]# ssh master
Last login: Fri Sep 9 17:42:39 2016 from localhost
[root@master ~]# hostname
master
[root@master ~]# exit
logout
Connection to master closed.
下载JDK,hadoop安装包,并配置环境变量自行百度。
4、配置hadoop 文件:
4.1、 Core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
<final>true></final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/urs/hadoop/tmp</value>
<description>A base for other temporary directories</description>
</property>
</configuration>
4.2、Hdfs-site.xml配置如下:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/usr/hadoop-1.2.1/name</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/hadoop-1.2.1/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>
</configuration>
4.3、mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>30.96.76.220:9001</value>
</property>
</configuration>
(注:如果后续访问 http://30.96.76.220:50070/ 失败。请关闭防火墙试一下。具体问题还是查看日志比较清楚。)
4.4、hadoop-env.sh:
export JAVA_HOME=/usr/jdk/jdk1.7.0_76
4.5 、masters和slaves文件
![](/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/51e990ea-79be-4b0b-866d-5399cedf0adf.png)
![](/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/4775f3ae-2a17-48eb-956c-a1b2de7ff7aa.png)
注:复制hoodoop 向 node1,node2.
5、格式化namenode
[root@master ~]# hadoop namenode -format
16/09/10 17:03:49 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/30.96.76.220
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_76
************************************************************/
16/09/10 17:03:49 INFO util.GSet: Computing capacity for map BlocksMap
16/09/10 17:03:49 INFO util.GSet: VM type = 64-bit
16/09/10 17:03:49 INFO util.GSet: 2.0% max memory = 1013645312
16/09/10 17:03:49 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/09/10 17:03:49 INFO util.GSet: recommended=2097152, actual=2097152
16/09/10 17:03:50 INFO namenode.FSNamesystem: fsOwner=root
16/09/10 17:03:50 INFO namenode.FSNamesystem: supergroup=supergroup
16/09/10 17:03:50 INFO namenode.FSNamesystem: isPermissionEnabled=true
16/09/10 17:03:50 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/09/10 17:03:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/09/10 17:03:50 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
16/09/10 17:03:50 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/10 17:03:50 INFO common.Storage: Image file /usr/hadoop-1.2.1/name/current/fsimage of size 110 bytes saved in 0 seconds.
16/09/10 17:03:51 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/hadoop-1.2.1/name/current/edits
16/09/10 17:03:51 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/hadoop-1.2.1/name/current/edits
16/09/10 17:03:51 INFO common.Storage: Storage directory /usr/hadoop-1.2.1/name has been successfully formatted.
16/09/10 17:03:51 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/30.96.76.220
************************************************************/
注意:上面只要出现“successfully formatted”就表示成功了。
6、启动hadoop
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/217b72ff-4f27-4ff6-8700-125c5432015f.png)
有这四项进程表示启动成功。
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/81e91112-cdbd-4e32-87f4-b4d37416775f.png)
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/e7423111-f0a3-4004-b9a6-573e82126afe.png)
表示node1.node2 启动成功
7、通过网站查看集群情况
![](https://note.wiz.cn/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/bebaa3ab-3c89-4f01-964f-6e257fd9a692.png)
![](/api/document/files/unzip/f2a88db2-23b3-4557-9ed8-d7833826b9fb/80e5df9f-44b1-4788-b9b6-6cff9b8f111c.562/index_files/0a22ed31-403d-4e0b-a12e-d525df9e1b2e.png)
全部成功。周六没有白加班。
最后附上原文连接: