hadoop(八)集群namenode启动ssh免密登录(完全分布式五)|10

前置章节：hadoop集群配置同步（hadoop完全分布式四）|10

启动namenode之前：

1. 先查看有无节点启动，执行jps查看，有的话停掉

[shaozhiqi@hadoop102 ~]$ jps
10566 Jps
[shaozhiqi@hadoop102 ~]$

2. 删除每个机器上之前运行的data和logs文件

启动namenode

格式化namenode
执行命令和执行成功的部分日志

[shaozhiqi@hadoop102 hadoop-3.1.2]$ bin/hdfs namenode –format
2019-06-25 06:07:55,932 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1212972932-192.168.1.102-1561468075913
2019-06-25 06:07:55,968 INFO common.Storage: Storage directory /opt/module/hadoop-3.1.2/data/tmp/dfs/name has been successfully formatted.
2019-06-25 06:07:55,988 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/module/hadoop-3.1.2/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2019-06-25 06:07:56,207 INFO namenode.FSImageFormatProtobuf: Image file /opt/module/hadoop-3.1.2/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds 
2019-06-25 06:07:56,237 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2019-06-25 06:07:56,260 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop102/192.168.1.102
************************************************************/
[shaozhiqi@hadoop102 hadoop-3.1.2]$

启动102的namenode,启动成功

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start namenode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement **"hdfs --daemon start"** instead.
[shaozhiqi@hadoop102 hadoop-3.1.2]$ ll
[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
10945 NameNode
11087 Jps
[shaozhiqi@hadoop102 hadoop-3.1.2]$

系统提示启动用hdfs --daemon start命令，可能是新版本更新了

启动102的datanode

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.

查看启动结果：

[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
10945 NameNode
11265 Jps
11205 DataNode
[shaozhiqi@hadoop102 hadoop-3.1.2]$

启动103.104的datanode

[shaozhiqi@hadoop103 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
[shaozhiqi@hadoop103 hadoop-3.1.2]$ jps
2816 DataNode
2833 Jps
[shaozhiqi@hadoop103 hadoop-3.1.2]$

启动 104的datanode

[shaozhiqi@hadoop104 module]$ cd hadoop-3.1.2/
[shaozhiqi@hadoop104 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
[shaozhiqi@hadoop104 hadoop-3.1.2]$ jps
2819 Jps
2805 DataNode
[shaozhiqi@hadoop104 hadoop-3.1.2]$

目前为止整个集群的namenode和datanode都启动了

在浏览器查看是否可以访问我们的namenode web页面，可以访问启动成功

访问namenode web页面

访问namenodeweb
访问namenodeweb

 

问题

假如我们的集群有100台机器每个都要这样起吗？

配置脚本ssh免密登录

文件解释
Know_hosts****记录ssh访问过的计算机的公钥
Id_ras****生成的私钥
Id_ras.pub****生成的公钥
Authorized_key****存放授权过的免密访问服务器的公钥

ssh命令可以直接登录到别的机器

[shaozhiqi@hadoop102 ~]$ ssh hadoop103
shaozhiqi@hadoop103's password:  
Last failed login: Tue Jun 25 06:41:19 PDT 2019 from hadoop102 on ssh:notty
There were 2 failed login attempts since the last successful login.
Last login: Tue Jun 25 06:03:37 2019 from 192.168.1.1
[shaozhiqi@hadoop103 ~]$ hostname
hadoop103
[shaozhiqi@hadoop103 ~]$

免密原理

免密登录原理图
免密登录原理图

到102找到 .ssh

[shaozhiqi@hadoop102 ~]$ ls -al
total 32
drwx------. 7 shaozhiqi shaozhiqi 4096 Jun 25 06:07 .
drwxr-xr-x. 3 root root 22 Jun 18 05:44 ..
-rw-------. 1 shaozhiqi shaozhiqi 1075 Jun 24 09:06 .bash_history
-rw-r--r--. 1 shaozhiqi shaozhiqi 18 Dec 6 2016 .bash_logout
-rw-r--r--. 1 shaozhiqi shaozhiqi 193 Dec 6 2016 .bash_profile
-rw-r--r--. 1 shaozhiqi shaozhiqi 231 Dec 6 2016 .bashrc
drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 24 08:36 bin
drwxrwxr-x. 3 shaozhiqi shaozhiqi 17 Jun 18 06:04 .cache
drwxrwxr-x. 3 shaozhiqi shaozhiqi 17 Jun 18 06:04 .config
drwxr-xr-x. 4 shaozhiqi shaozhiqi 37 May 4 2017 .mozilla
drwx-----. 2 shaozhiqi shaozhiqi 24 Jun 22 23:04 .ssh
-rw-------. 1 shaozhiqi shaozhiqi 6420 Jun 25 06:07 .viminfo
-rw-------. 1 shaozhiqi shaozhiqi 165 Jun 24 16:42 .Xauthority
[shaozhiqi@hadoop102 ~]$

进入.ssh生成公钥和私钥

[shaozhiqi@hadoop102 ~]$ cd .ssh/
[shaozhiqi@hadoop102 .ssh]$ ls
known_hosts
[shaozhiqi@hadoop102 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/shaozhiqi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/shaozhiqi/.ssh/id_rsa.
Your public key has been saved in /home/shaozhiqi/.ssh/id_rsa.pub.
The key fingerprint is:
76:b4:c5:20:b4:bb:ef:42:81:1f:60:98:1f:56:40:c1 shaozhiqi@hadoop102
The key's randomart image is:
+--[ RSA 2048]----+

| =+=+ . |

| o E o o |

| + +. . o |

| o oo o |

+-----------------+
[shaozhiqi@hadoop102 .ssh]$ ll
total 12
-rw-------. 1 shaozhiqi shaozhiqi 1675 Jun 25 06:52 id_rsa
-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 06:52 id_rsa.pub
-rw-r--r--. 1 shaozhiqi shaozhiqi 370 Jun 23 06:05 known_hosts
[shaozhiqi@hadoop102 .ssh]$

查看生成的公钥

[shaozhiqi@hadoop102 .ssh]$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102
[shaozhiqi@hadoop102 .ssh]$

将公钥copy到103

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop103
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
shaozhiqi@hadoop103's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop103'"
and check to make sure that only the key(s) you wanted were added.
[shaozhiqi@hadoop102 .ssh]$

查看103,验证是否copy成功

[shaozhiqi@hadoop103 ~]$ cd .ssh/
[shaozhiqi@hadoop103 .ssh]$ pwd
/home/shaozhiqi/.ssh
[shaozhiqi@hadoop103 .ssh]$ ls
authorized_keys known_hosts
[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102
[shaozhiqi@hadoop103 .ssh]$

同样将公钥copy到104

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop104
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
shaozhiqi@hadoop104's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop104'"
and check to make sure that only the key(s) you wanted were added.
[shaozhiqi@hadoop102 .ssh]$

这样以后在存102到103和104是可以免密跳转的。但是往回跳跳102还是需要输入密码

[shaozhiqi@hadoop102 .ssh]$ ssh hadoop103
Last login: Tue Jun 25 06:41:28 2019 from hadoop102
[shaozhiqi@hadoop103 ~]$ ssh hadoop102
shaozhiqi@hadoop102's password:
Last login: Tue Jun 25 06:43:20 2019 from hadoop103
[shaozhiqi@hadoop102 ~]$

往回跳可以用exit直接回去

[shaozhiqi@hadoop102 ~]$ ssh hadoop103
Last login: Tue Jun 25 07:04:23 2019 from hadoop102
[shaozhiqi@hadoop103 ~]$ exit
logout
Connection to hadoop103 closed.
[shaozhiqi@hadoop102 ~]$

后来发现访问自己也需要输入密码，所以啊，还得把公钥copy给自己一份

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop102
[shaozhiqi@hadoop102 .ssh]$ ll
total 16
-rw-------. 1 shaozhiqi shaozhiqi 401 Jun 25 07:11 authorized_keys
-rw-------. 1 shaozhiqi shaozhiqi 1675 Jun 25 06:52 id_rsa
-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 06:52 id_rsa.pub
-rw-r--r--. 1 shaozhiqi shaozhiqi 555 Jun 25 07:11 known_hosts
[shaozhiqi@hadoop102 .ssh]$

为啥

要让102可以免密访问其他机器呢？因为102的有我们的namenode他需要向其他机器分发和通信
同样103机器有我们的resoucemanager,也许要去控制访问其他节点的信息，所以在103上也要去访问别的机器哈哈。。。

生成103的公钥

[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102
[shaozhiqi@hadoop103 .ssh]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/shaozhiqi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/shaozhiqi/.ssh/id_rsa.
Your public key has been saved in /home/shaozhiqi/.ssh/id_rsa.pub.
The key fingerprint is:
6b:da:d0:d2:ab:8f:ff:48:bb:47:04:0a:8d:28:dc:c3 shaozhiqi@hadoop103
The key's randomart image is:
+--[ RSA 2048]----+
|. o. o |
|..Eo . . |
| . .. . . |
| O +. |
| ++O+. |
+-----------------+
[shaozhiqi@hadoop103 .ssh]$ ll
total 16
-rw-------. 1 shaozhiqi shaozhiqi 401 Jun 25 06:58 authorized_keys
-rw-------. 1 shaozhiqi shaozhiqi 1679 Jun 25 07:16 id_rsa
-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 07:16 id_rsa.pub
-rw-r--r--. 1 shaozhiqi shaozhiqi 370 Jun 22 23:03 known_hosts
[shaozhiqi@hadoop103 .ssh]$

将公钥同步到102,103,104

[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop102
。。。
[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop103
。。。
[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop104

查看是否copy成功，可以看到103多了一个103的公钥

[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpI16PazeKVEHHmjRuHtkjT8jl3d+3Ra7+wzUA6Xtiky/BV3zQO64oLEOi7tSkGVPgupOWT5yGXhILJFYEKvCp91rP/pQWQFPNFnX+MbM6iTBDO8eL99Ma6owcipuVCGVGw5bPnX4F+4GZSOeCIsd21pJ4CU0OjdY64IneIdRokoMDmqvLzgT+/Hjb4c+HPzx4OUwcQyzN9UrWWsL557ahVfjwickdkLgR+33aTK5IOl8XCtkuMNJxqRzrsqgPKy01f+psdRBQkX0qCO6laL5laq/7zPmTdz9JZC4rekjYa0cTbvInX1C6wV+NTaedzRs2ewoGpu/VsU17eIWdDuLR shaozhiqi@hadoop103
[shaozhiqi@hadoop103 .ssh]$

同样的去查看其他两台机器都是copy成功

验证从103跳102,104都是ok

[shaozhiqi@hadoop103 .ssh]$ ssh hadoop102
Last login: Tue Jun 25 07:04:46 2019 from hadoop103
[shaozhiqi@hadoop102 ~]$
…

102上后续需要部分root权限操作我们hadoop基群，所以也要给root用户配置免密登录，当前只实现shaozhiqi用户

继续操作在102 切到root

[shaozhiqi@hadoop102 .ssh]$ su root
Password:
ABRT has detected 2 problem(s). For more info run: abrt-cli list --since 1561257992
[root@hadoop102 .ssh]#

发现我的102尽然没有.ssh

[root@hadoop102 ~]# cd .ssh/
bash: cd: .ssh/: No such file or directory
[root@hadoop102 ~]#

执行这个命令登录一次就有了

[root@hadoop102 ~]# ssh localhost

验证

[root@hadoop102 ~]# cd .ssh
[root@hadoop102 .ssh]# ll
total 4
-rw-r--r--. 1 root root 171 Jun 25 07:34 known_hosts
[root@hadoop102 .ssh]#

生成102 root的公钥私钥

[root@hadoop102 .ssh]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
95:24:14:30:a6:9d:3d:37:6d:46:19:08:35:45:72:1d root@hadoop102
The key's randomart image is:
+--[ RSA 2048]----+
| +o=+=+*+E. |
| + + o.B. . |
| . o o = + |
| |
+-----------------+
[root@hadoop102 .ssh]#

同步root公钥到102，103，104

[root@hadoop102 .ssh]# ssh-copy-id hadoop102
。。。
[root@hadoop102 .ssh]# ssh-copy-id hadoop103
。。。
[root@hadoop102 .ssh]# ssh-copy-id hadoop104
。。。

验证ok

[root@hadoop102 .ssh]# ssh hadoop103
Last login: Tue Jun 25 07:32:53 2019
[root@hadoop103 ~]#

总结下

  1. 102的shaozhiqi/root可以免密登录到102,103,104
  2. 103的shaozhiqi可以免密登录到102,103,104
  3. 这一节在102启动了namenode
  4. 在102,103,104启动了datanode

下一节计划启动集群 no time do more

posted @ 2019-09-17 16:18  ~清风煮酒~  阅读(1280)  评论(0)  编辑  收藏  举报