面向生产环境的大集群模式安装Hadoop
一、实验说明
1、本实验将使用DNS而不是hosts文件解析主机名；
2、使用NFS共享密钥文件，而不是逐个手工拷贝添加密钥；
3、复制Hadoop时使用批量拷贝脚本而不是逐台复制。
测试环境：
| Hostname | IP | Hadoop版本 | Hadoop | 功能 | 系统 |
| hadoop1 | 192.168.1.161 | 0.20.0 | namenode | nfs服务器端 | rhel5.4x86 |
| hadoop2 | 192.168.1.162 | 0.20.0 | datanode | dns+nfs客服端 | rhel5.4 x86 |
| hadoop3 | 192.168.1.163 | 0.20.0 | datanode | nfs客户端 | rhel5.4 x86 |
二、DNS的安装与配置
1、上传dns目录：
1 [root@hadoop2 dns]# ls 2 dnsmasq.conf dnsmasq.hosts dnsmasq.resolv.conf pid start.sh stop.sh
2、修改dns目录中的文件：
----dnsmasq.conf为dnsmasq的配置文件----
[root@hadoop2 dns]# cat dnsmasq.conf cache-size=50000 dns-forward-max=1000 resolv-file=/dns/dnsmasq.resolv.conf addn-hosts=/dns/dnsmasq.hosts ----dnsmasq缓存下来的域名，不使用/etc/hosts----
[root@hadoop2 dns]# cat dnsmasq.hosts 192.168.1.161 hadoop1 192.168.1.162 hadoop2 192.168.1.163 hadoop3 ----在dnsmasq.resolv.conf添加上游dns的地址----
[root@hadoop2 dns]# cat dnsmasq.resolv.conf ### /etc/resolv.conf file autogenerated by netconfig! # # Before you change this file manually, consider to define the # static DNS configuration using the following variables in the # /etc/sysconfig/network/config file: # NETCONFIG_DNS_STATIC_SEARCHLIST # NETCONFIG_DNS_STATIC_SERVERS # NETCONFIG_DNS_FORWARDER # or disable DNS configuration updates via netconfig by setting: # NETCONFIG_DNS_POLICY='' # # See also the netconfig(8) manual page and other documentation. # # Note: Manual change of this file disables netconfig too, but # may get lost when this file contains comments or empty lines # only, the netconfig settings are same with settings in this # file and in case of a "netconfig update -f" call. # nameserver 218.108.248.228
nameserver 218.108.248.200
[root@hadoop2 dns]# cat start.sh #!/bin/sh killall dnsmasq dnsmasq --port=53 --pid-file=/dns/pid --conf-file=/dns/dnsmasq.conf [root@hadoop2 dns]# cat stop.sh #!/bin/sh killall dnsmasq
3、启动dns，并在hadoop2上进行测试：
[root@hadoop2 dns]# dig @hadoop2 www.qq.com ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> @hadoop2 www.qq.com ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41272 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 7 ;; QUESTION SECTION: ;www.qq.com. IN A ;; ANSWER SECTION: www.qq.com. 5 IN A 182.254.8.146 ;; AUTHORITY SECTION: www.qq.com. 5 IN NS ns-cmn1.qq.com. www.qq.com. 5 IN NS ns-cnc1.qq.com. www.qq.com. 5 IN NS ns-os1.qq.com. ;; ADDITIONAL SECTION: ns-os1.qq.com. 5 IN A 184.105.66.196 ns-os1.qq.com. 5 IN A 202.55.2.226 ns-os1.qq.com. 5 IN A 202.55.2.230 ns-os1.qq.com. 5 IN A 114.134.85.106 ns-cmn1.qq.com. 5 IN A 120.204.202.200 ns-cnc1.qq.com. 5 IN A 125.39.127.27 ns-cnc1.qq.com. 5 IN A 61.135.167.182 ;; Query time: 33 msec ;; SERVER: 192.168.1.162#53(192.168.1.162) ;; WHEN: Sat Aug 24 19:40:25 2013 ;; MSG SIZE rcvd: 221
4、配置hadoop1和hadoop3的/etc/resolv.conf
----dns客户端需要对/etc/resolv.conf文件中注释掉search localdomain，否则无法解析dnsmasq缓存下来的域名----
[hadoop@hadoop3 ~]$ cat /etc/resolv.conf ; generated by /sbin/dhclient-script #search localdomain #nameserver 192.168.11.2
nameserver 192.168.1.162 [root@hadoop3 ~]# dig @hadoop2 www.qq.com ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> @hadoop2 www.qq.com ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2061 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 7 ;; QUESTION SECTION: ;www.qq.com. IN A ;; ANSWER SECTION: www.qq.com. 5 IN A 182.254.8.146 ;; AUTHORITY SECTION: www.qq.com. 5 IN NS ns-cnc1.qq.com. www.qq.com. 5 IN NS ns-cmn1.qq.com. www.qq.com. 5 IN NS ns-os1.qq.com. ;; ADDITIONAL SECTION: ns-os1.qq.com. 5 IN A 202.55.2.226 ns-os1.qq.com. 5 IN A 202.55.2.230 ns-os1.qq.com. 5 IN A 114.134.85.106 ns-os1.qq.com. 5 IN A 184.105.66.196 ns-cmn1.qq.com. 5 IN A 120.204.202.200 ns-cnc1.qq.com. 5 IN A 61.135.167.182 ns-cnc1.qq.com. 5 IN A 125.39.127.27 ;; Query time: 24 msec ;; SERVER: 192.168.1.162#53(192.168.1.162) ;; WHEN: Sat Aug 24 19:44:43 2013 ;; MSG SIZE rcvd: 221
三、配置NFS
1、查看nfs是否已经安装
[root@hadoop1 ~]# rpm -qa |grep nfs nfs-utils-1.0.9-42.el5 nfs-utils-lib-1.0.8-7.6.el5
2、编辑/etc/exports
[root@hadoop1 ~]# cat /etc/exports /home/hadoop/.ssh/ *(rw,sync,no_root_squash)
3、创建hadoop用户
[root@hadoop1 ~]# useradd hadoop [root@hadoop1 ~]# passwd hadoop Changing password for user hadoop. New UNIX password: BAD PASSWORD: it is based on a dictionary word Retype new UNIX password: passwd: all authentication tokens updated successfully.
4、生成ssh密钥
[hadoop@hadoop1 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 11:a6:28:73:db:0b:c2:47:fc:c9:8d:1c:0c:b4:6e:00 hadoop@hadoop1
5、修改挂载点的属性
[root@hadoop1 ~]# chmod 777 /home/hadoop/.ssh/
6、重启nfs
[root@hadoop1 ~]# service nfs restart Shutting down NFS mountd: [FAILED] Shutting down NFS daemon: [FAILED] Shutting down NFS quotas: [FAILED] Shutting down NFS services: [FAILED] Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS daemon: [ OK ] Starting NFS mountd: [ OK ]
7、在本机挂载测试
[root@hadoop1 ~]# mount 192.168.1.161:/home/hadoop/.ssh /mnt [root@hadoop1 ~]# mount /dev/sda1 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) 192.168.1.161:/home/hadoop/.ssh on /mnt type nfs (rw,addr=192.168.1.161) [root@hadoop1 ~]# ll /home/hadoop/.ssh/ total 8 -rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa -rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub [root@hadoop1 ~]# ll /mnt total 8 -rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa -rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub
四、nfs整合ssh密钥
1、先将id_rsa.pub拷贝成authorized_keys
[hadoop@hadoop1 ~]$ cp .ssh/id_rsa.pub .ssh/authorized_keys
2、再登陆hadoop2和hadoop3创建hadoop用户并用hadoop登陆，然后生成每个机器的ssh的rsa密钥
----hadoop2和hadoop3操作一样----
[root@hadoop2 dns]# useradd hadoop [root@hadoop2 dns]# passwd hadoop Changing password for user hadoop. New UNIX password: BAD PASSWORD: it is based on a dictionary word Retype new UNIX password: passwd: all authentication tokens updated successfully. [root@hadoop2 dns]# su - hadoop [hadoop@hadoop2 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 3c:9d:07:2a:7d:3d:e3:d3:22:0c:0e:8b:5d:96:93:e1 hadoop@hadoop2
3、在hadoop2和hadoop3上挂载nfs
[root@hadoop2 dns]# mount 192.168.1.161:/home/hadoop/.ssh /mnt [root@hadoop2 dns]# ll /mnt total 12 -rw-r--r-- 1 hadoop hadoop 396 Aug 25 11:04 authorized_keys -rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa -rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub [root@hadoop2 dns]# mount /dev/sda1 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 192.168.1.161:/home/hadoop/.ssh on /mnt type nfs (rw,addr=192.168.1.161)
4、把hadoop2和hadoop3的公钥id_rsa_pub都添加到/mnt/authorized_keys里
[root@hadoop2 dns]# cat /home/hadoop/.ssh/id_rsa.pub >> /mnt/authorized_keys [root@hadoop3 ~]# cat /home/hadoop/.ssh/id_rsa.pub >> /mnt/authorized_keys
5、查看authorized_keys内容
[hadoop@hadoop1 ~]$ cat /mnt/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA32vNwXv/23k0yF7QeITb61J5uccudHB3gQBtqCnB7wsOtIhdUsVIfcxGmPnWp6S9V+Ob+b73Vrl2xsxP4i0N8Cu1l2ZcU9jevc+o37yX4nW2oTBFVEP31y9E9fXkYf3cKiF0UrvunL59qgNnVUbq8qRtFr5QPAx6lGY0TYZiPaPr+POwNKF1IZvToqABsOnNimv0DNmAhbd3QyM7GaR/ZRQKOCMF8NYljo6exoDk9xPq/wCHC/rBnAU3gUlwi7Kn/tk2dirwvYZuqP3VO+w5zd6sYxscD8+UNK99XdOARzTlc8/iEPHy+JSBa6sQI2hOAOCAuHBtTymoJFUDH9YqXQ== hadoop@hadoop1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4lTx6JTZlhoLI4Yyo0a6YeDmIgz60pYwYKwVL+p4wfp9OWB2/sEyf9iCsK8i94mnWMfNsRehqAG2ucPmWz1s/Kufxu/6uc8hJjDlOOMUOE7ENyN0Zre5MHj8jauDRhY4y37Rh3Crx86wzq79isDqJOWnKyjPQDjUH45780Hvtk87ckwNNSFhwuRgTFKhz0bQloJuHazU1/W924wmicqeEUSGhUFEkXUeJu7FqQjJcPjoRNqyTEuCHiYVh9HjOrUPdosfYqmQfuZ/x2gmsGRUdfTl32rkoZW43ay8CFV/MKqAFucEOiiHW7xttmm3zJgcyLptGhjo7NtvAQwKkPfG6w== hadoop@hadoop2 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs7fkzQMR6yVqLBVAnJqTxFPO9NNngrmYDNZMbWXDz6V8J4Z7zC46odUERe3CNjC+v3X8rwvUWlALYtvMNonQwhnpvqe2s0CpDithSFkOt5fQarRYP5JtAjHvF5b22NqcyltF+ywLT4zKAg4tjgGV5nLafI2hsNjgljUOXkRjpwSSUpLmLayWnepLIwioCPPGIkM40balUOEWEASzaI4DaPoywmoVUrByou71i1F1VizXpbhIWW+LE2cANAy1xmP0zYBa+/O4mvpgZjWLtLpKFR/1nRZPh1emy+OB6RcoJl3Awmhcsyyjd4Q8jfOYsH78PKpnwJfyhtUEIENrzUV63w== hadoop@hadoop3
6、做软连接(此步骤不需要在nfs服务器端做，只在客户端做)
[hadoop@hadoop2 ~]$ ln -s /mnt/authorized_keys /home/hadoop/.ssh/authorized_keys [hadoop@hadoop2 ~]$ ll /home/hadoop/.ssh/authorized_keys lrwxrwxrwx 1 hadoop hadoop 20 Aug 25 11:14 /home/hadoop/.ssh/authorized_keys -> /mnt/authorized_keys [hadoop@hadoop3 ~]$ ln -s /mnt/authorized_keys /home/hadoop/.ssh/authorized_keys [hadoop@hadoop3 ~]$ ll /home/hadoop/.ssh/authorized_keys lrwxrwxrwx 1 hadoop hadoop 20 Aug 25 11:15 /home/hadoop/.ssh/authorized_keys -> /mnt/authorized_keys
7、修改权限
[hadoop@hadoop1 ~]$ chmod 700 /home/hadoop/.ssh/
备注：如果不修改的话，在进行登陆的时候会出现需要密码。
8、测试是否实验无密码登陆
[hadoop@hadoop1 ~]$ ssh hadoop2 The authenticity of host 'hadoop2 (192.168.1.162)' can't be established. RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hadoop2,192.168.1.162' (RSA) to the list of known hosts. [hadoop@hadoop2 ~]$ ssh hadoop3 The authenticity of host 'hadoop3 (192.168.1.163)' can't be established. RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hadoop3,192.168.1.163' (RSA) to the list of known hosts. [hadoop@hadoop3 ~]$
五、批量安装Hadoop
1、先在hadoop1上把namenode安装完成，安装hadoop分布式可以参考：Hadoop集群安装
[hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp hadoop-0.20.2 hadoop@"$1":/home/hadoop/"}' > scp.sh [hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp temp hadoop@"$1":/home/hadoop/"}' >> scp.sh [hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp user hadoop@"$1":/home/hadoop/"}' >> scp.sh [hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp jdk1.7 hadoop@"$1":/home/hadoop/"}' >> scp.sh [hadoop@hadoop1 ~]$ ls hadoop-0.20.2 jdk1.7 scp.sh temp user [hadoop@hadoop1 ~]$ cat scp.sh scp -rp hadoop-0.20.2 hadoop@192.168.1.162:/home/hadoop/ scp -rp hadoop-0.20.2 hadoop@192.168.1.163:/home/hadoop/ scp -rp temp hadoop@192.168.1.162:/home/hadoop/ scp -rp temp hadoop@192.168.1.163:/home/hadoop/ scp -rp user hadoop@192.168.1.162:/home/hadoop/ scp -rp user hadoop@192.168.1.163:/home/hadoop/ scp -rp jdk1.7 hadoop@192.168.1.162:/home/hadoop/ scp -rp jdk1.7 hadoop@192.168.1.163:/home/hadoop/
2、格式化namenode
[hadoop@hadoop1 ~]$ hadoop-0.20.2/bin/hadoop namenode -format 13/08/25 11:52:39 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop1/192.168.1.161 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 0.20.2 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 ************************************************************/ Re-format filesystem in /home/hadoop/user/name ? (Y or N) Y 13/08/25 11:52:46 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop 13/08/25 11:52:46 INFO namenode.FSNamesystem: supergroup=supergroup 13/08/25 11:52:46 INFO namenode.FSNamesystem: isPermissionEnabled=true 13/08/25 11:52:47 INFO common.Storage: Image file of size 96 saved in 0 seconds. 13/08/25 11:52:48 INFO common.Storage: Storage directory /home/hadoop/user/name has been successfully formatted. 13/08/25 11:52:48 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop1/192.168.1.161 ************************************************************/
3、启动hadoop
[hadoop@hadoop1 ~]$ hadoop-0.20.2/bin/start-all.sh starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-hadoop1.out 192.168.1.163: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-hadoop3.out 192.168.1.162: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-hadoop2.out The authenticity of host '192.168.1.161 (192.168.1.161)' can't be established. RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a. Are you sure you want to continue connecting (yes/no)? yes 192.168.1.161: Warning: Permanently added '192.168.1.161' (RSA) to the list of known hosts. 192.168.1.161: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-hadoop1.out starting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-hadoop1.out 192.168.1.162: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-hadoop2.out 192.168.1.163: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-hadoop3.out
4、查看各个节点
[hadoop@hadoop1 ~]$ jdk1.7/bin/jps 4416 Jps 4344 JobTracker 4306 SecondaryNameNode 4157 NameNode [hadoop@hadoop2 ~]$ jdk1.7/bin/jps 3699 TaskTracker 3636 DataNode 3752 Jps [hadoop@hadoop3 ~]$ jdk1.7/bin/jps 4763 TaskTracker 4834 Jps 4653 DataNode
六、重点说明
1、如果重启以后无法自动挂载nfs，可以在/etc/rc.d/rc.local文件中添加：
/bin/mount -a
2、如果IP是自动获取的，请在DNS主机的/etc/rc.d/rc.local文件添加：
/bin/cat /app/resolv.conf > /etc/resolv.conf
[root@node1 ~]# cat /app/resolv.conf ; generated by /sbin/dhclient-script #search localdomain #nameserver 192.168.1.151
其它主机的/etc/rc.d/rc.local添加：
/bin/cat /app/resolv.conf > /etc/resolv.conf
[root@node2 ~]# cat /app/resolv.conf ; generated by /sbin/dhclient-script #search localdomain nameserver 192.168.1.151
