五、Hadoop伪分布式

5.1 伪分布式介绍
–伪分布式的安装和完全分布式类似，但区别是所有角色安装在一台机器上，使用本地磁盘，一般生产环境都会使用完全分布式，伪分布式一般用来学习和测试方面的功能
–伪分布式的配置和完全分布式配置类似的词
5.2 配置文件格式
–Hadoop-env.sh
JAVA_HOME
HADOOP_CONF_DIR
–Xml 文件配置格式
<property>
    <name>关键字</name>
    <value>变量值</value>
    <description> 描述</description>
</property>
5.4 HDFS 分布式文件系统
5.4.1 系统规划

主机	角色	软件
192.168.0.10（Nn）	NameNode SecondaryNameNode	HDFS
192.168.0.11（Node1）	DataNode	HDFS
192.168.0.12（Node2）	DataNode	HDFS
192.168.0.13（Node3）	DataNode	HDFS

 

5.4.2 基础环境准备
–新开启3台虚拟机
–禁用selinux
SELINUX=disabled
–禁用firewalld
systemctlstop firewalld
systemctlmask firewalld
–安装java-1.8.0-openjdk-devel
yum install -y java-1.8.0-openjdk-devel
–在3台机器上配置/etc/hosts (--ALL)
[root@node1 ~]# vim /etc/hosts
192.168.0.10 nn01
192.168.0.11 node1
192.168.0.12 node2
192.168.0.21 node3
–注：所有主机都能ping 同namenode的主机名(--ALL)
–namenode能ping 同所有节点
[root@node1 ~]# ping node2
PING node2 (192.168.0.12) 56(84) bytes of data.
64 bytes from node2 (192.168.0.12): icmp_seq=1 ttl=64 time=1.41 ms
64 bytes from node2 (192.168.0.12): icmp_seq=2 ttl=64 time=0.576 ms
–java -version 验证java 安装 (--ALL)
[root@node1 ~]# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
–jps验证角色
[root@node1 ~]# jps
864 Jps
5.4.3 配置SSH 信仸关系（nn01）
–注意：不能出现要求输入yes 的情况，每台机器都要能登录成功，包括本机！！！
–ssh_config
StrictHostKeyChecking no
–ssh-keygen -b 2048 -t rsa-N '' -f key
–ssh-copy-id -i ./key.pub root@ip.xx.xx.xx
eg:
[root@nn01 ~]# vim /etc/ssh/ssh_config
58 Host *
59 GSSAPIAuthentication yes
60 StrictHostKeyChecking no #追加项
[root@nn01 ~]# ssh-keygen -b 2048 -t rsa -N '' -f key
[root@nn01 ~]# cd .ssh/
[root@nn01 .ssh]# ssh-copy-id -i id_rsa.pub nn01
[root@nn01 .ssh]# ssh-copy-id -i id_rsa.pub node1（注node1-node3）
5.4.4 HDFS 完全分布式系统配置
[root@nn01 etc]# cd /usr/local/hadoop/etc/hadoop
①–环境配置文件hadoop-env.sh
[root@nn01 hadoop]# vim hadoop-env.sh
25 export JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre"
33 export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
②–核心配置文件core-site.xml
模板：（关键字、变量、描述可以在官网上查找参考5.5）
<property>
<name>关键字</name>
<value>变量值</value>
<description> 描述</description> #可省略
</property>
[root@nn01 hadoop]# vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://nn01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop</value>
</property>
</configuration>

③–HDFS配置文件hdfs-site.xml
[root@nn01 hadoop]# vim hdfs-site.xmls
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>nn01:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>nn01:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

④–节点配置文件slaves
[root@nn01 hadoop]# vim slaves
node1
node2
node3
–同步配置
–hadoop所有节点的配置参数完全一样，我们在一台配置好以后，要把配置文件分发到其它所有主机上去
for i in node{1..3}; do scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh $i://usr/local/hadoop/etc/hadoop; scp /usr/local/hadoop/etc/hadoop/core-site.xml $i://usr/local/hadoop/etc/hadoop; scp /usr/local/hadoop/etc/hadoop/hdfs-site.xml $i://usr/local/hadoop/etc/hadoop; scp /usr/local/hadoop/etc/hadoop/slaves $i://usr/local/hadoop/etc/hadoop; done
–在所有机器上创建/var/hadoop文件夹
[root@nn01 hadoop]# mkdir /var/hadoop
[root@nn01 hadoop]# ssh node1 mkdir /var/hadoop
[root@nn01 hadoop]# ssh node2 mkdir /var/hadoop
[root@nn01 hadoop]# ssh node3 mkdir /var/hadoop
–在nn01上执行格式化操作
[root@nn01 ~]# cd /usr/local/hadoop/
[root@nn01 hadoop]# ./bin/hdfs namenode -format
21/08/27 15:04:31 INFO common.Storage: Storage directory /var/hadoop/dfs/name has been successfully formatted.
出现上述信息表示格式格式化成功
–启动集群
[root@nn01 hadoop]# ./sbin/start-dfs.sh
Starting namenodes on [nn01]
nn01: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-nn01.out
node2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node2.out
node1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node1.out
node3: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node3.out
Starting secondary namenodes [nn01]
nn01: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-nn01.out
执行完成以后会自动生成一个logs的文件夹
停止服务：[root@nn01 hadoop]# ./sbin/stop-dfs.sh
5.5 JPS验证角色：
[root@nn01 hadoop]# jps
2274 Jps
1956 NameNode
2143 SecondaryNameNode
[root@nn01 hadoop]# ssh node1 jps
1351 Jps
1274 DataNode
[root@nn01 hadoop]# ssh node2 jps
1204 Jps
1126 DataNode
[root@nn01 hadoop]# ssh node3 jps
1217 Jps
1139 DataNode

5.6 节点验证：
[root@nn01 hadoop]# ./bin/hdfs dfsadmin -report
Configured Capacity: 51505004544 (47.97 GB)
Present Capacity: 45111828480 (42.01 GB)
DFS Remaining: 45111803904 (42.01 GB)
DFS Used: 24576 (24 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 192.168.0.21:50010 (node3)
Hostname: node3
Decommission Status : Normal
Configured Capacity: 17168334848 (15.99 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 2149875712 (2.00 GB)
DFS Remaining: 15018450944 (13.99 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.48%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Aug 27 15:39:23 CST 2021

Name: 192.168.0.12:50010 (node2)
Hostname: node2
Decommission Status : Normal
Configured Capacity: 17168334848 (15.99 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 2147782656 (2.00 GB)
DFS Remaining: 15020544000 (13.99 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.49%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Aug 27 15:39:21 CST 2021

Name: 192.168.0.11:50010 (node1)
Hostname: node1
Decommission Status : Normal
Configured Capacity: 17168334848 (15.99 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 2095517696 (1.95 GB)
DFS Remaining: 15072808960 (14.04 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.79%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Aug 27 15:39:24 CST 2021
5.6 拍错步骤
[root@nn01 logs]# ls -l
拍namenode的错：
-rw-r--r-- 1 root root 31192 8月 27 15:26 hadoop-root-namenode-nn01.log
-rw-r--r-- 1 root root 714 8月 27 15:25 hadoop-root-namenode-nn01.out
排secondarynamenode的错
-rw-r--r-- 1 root root 23436 8月 27 15:26 hadoop-root-secondarynamenode-nn01.log
-rw-r--r-- 1 root root 714 8月 27 15:25 hadoop-root-secondarynamenode-nn01.out
通用
-rw-r--r-- 1 root root 0 8月 27 15:25 SecurityAuth-root.audit
如果排DataNode需要去node1-node3下的logs下查看
[root@node1 logs]# ls -l
总用量 28
-rw-r--r-- 1 root root 22695 8月 27 15:25 hadoop-root-datanode-node1.log
-rw-r--r-- 1 root root 714 8月 27 15:25 hadoop-root-datanode-node1.out
-rw-r--r-- 1 root root 0 8月 27 15:25 SecurityAuth-root.audit
5.5 官网文件手册的查看（http://hadoop.apache.org/）选择Documentation--->在选择你的版本--->进入版本后--->Configuration(中有我们具体的配置文件文件介绍)