1. 集群规划：
192.168.1.252 palo252 Namenode+Datanode 192.168.1.253 palo253 YarnManager+Datanode+SecondaryNameNode 192.168.1.254 palo254 Datanode
2. 设定固定IP地址
vi /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet BOOTPROTO=static DEFROUTE=yes NAME=eth0 UUID=7ac09286-c35b-4f15-a9ba-701c093832bf DEVICE=eth0 IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_PRIVACY=no ONBOOT=yes DNS1=192.168.1.1 IPADDR=192.168.1.252 #三台机器都要分别设置 PREFIX=24 GATEWAY=192.168.1.1
3. 修改主机名：
192.168.1.252
hostnamectl set-hostname palo252 hostnamectl --static set-hostname palo252
192.168.1.253
hostnamectl set-hostname palo253 hostnamectl --static set-hostname palo253
192.168.1.254
hostnamectl set-hostname palo254 hostnamectl --static set-hostname palo254
4. 修改hosts文件
vi /etc/hosts
127.0.0.1 localhost ::1 localhost 192.168.1.252 palo252 192.168.1.253 palo253 192.168.1.254 palo254
5. 安装JDK(所有节点）
具体到oracle官网下载
6. SSH免密登录
Precondition: install ssh server if not avalible
#install ssh client and ssh-server sudo yum install -y openssl openssh-server #enable ssh server to start at system start up systemctl enable sshd.service #start ssh server service systemctl start sshd.service
A） 每台机器生成访问秘钥，复制到192.168.1.252:/home/workspace目录下
192.168.1.252：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cp ~/.ssh/authorized_keys /home/workspace/authorized_keys252 rm -rf ~/.ssh/authorized_keys #删除公钥文件
192.168.1.253：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys scp ~/.ssh/authorized_keys 192.168.1.252:/home/workspace/authorized_keys253 rm -rf ~/.ssh/authorized_keys #删除公钥文件
192.168.1.254：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys scp ~/.ssh/authorized_keys 192.168.1.252:/home/workspace/authorized_keys254 rm -rf ~/.ssh/authorized_keys #删除公钥文件
B） 在192.168.1.252上将所有的公钥合并成一个公钥文件
cat /home/workspace/authorized_keys252 >> /home/workspace/authorized_keys cat /home/workspace/authorized_keys253 >> /home/workspace/authorized_keys cat /home/workspace/authorized_keys254 >> /home/workspace/authorized_keys
C） 将合并后的公钥文件复制到集群中的各个主机中
scp /home/workspace/authorized_keys 192.168.1.253:~/.ssh/ scp /home/workspace/authorized_keys 192.168.1.254:~/.ssh/ cp /home/workspace/authorized_keys ~/.ssh/ #因为目前在252主机中，所以使用的命令为cp而不是scp
注：也可以借助 ssh-copy-id -i ~/.ssh/id_rsa.pub {ip or hostname}来往远程机器复制公钥
以本集群的配置为例，以上ABC三步的操作亦可以通过下面的操作来完成，操作方法如下：
192.168.1.252，192.168.1.253，192168.1.254 均做以下操作，就完成了私钥的生成，公钥的分发
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa #生成本机公钥和私钥 ssh-copy-id -i ~/.ssh/id_rsa.pub palo252 #复制本机的公钥到palo252机器上，默认会存储在远程机器的~/.ssh/authorized_keys文件中，如果此文件不存在，会创建该文件 ssh-copy-id -i ~/.ssh/id_rsa.pub palo253 #复制本机的公钥到palo252机器上，默认会存储在远程机器的~/.ssh/authorized_keys文件中，如果此文件不存在，会创建该文件 ssh-copy-id -i ~/.ssh/id_rsa.pub palo254 #复制本机的公钥到palo252机器上，默认会存储在远程机器的~/.ssh/authorized_keys文件中，如果此文件不存在，会创建该文件
D） 每台机器：
chmod 755 ~ #当前用户根目录访问权限 chmod 700 ~/.ssh/ #.ssh目录权限 chmod 600 ~/.ssh/id_rsa #id_rsa的访问权限 chmod 644 ~/.ssh/id_rsa.pub #id_rsa.pub的访问权限 chmod 644 ~/.ssh/authorized_keys #authorized_keys的访问权限
说明：
如果ssh 登录的时候失败或者需要密码才能登陆，可以查看sshd的日志信息。日志信息目录为，/var/log/secure
你会发现如下字样的日志信息。
Jul 22 14:20:33 v138020.go sshd[4917]: Authentication refused: bad ownership or modes for directory /home/edw
则需要设置权限：sshd为了安全，对属主的目录和文件权限有所要求。如果权限不对，则ssh的免密码登陆不生效。
用户目录权限为 755 或者 700，就是不能是77x。
.ssh目录权限一般为755或者700。
rsa_id.pub 及authorized_keys权限一般为644
rsa_id权限必须为600
可通过来查看ssh过程中的日志.
cat /var/log/secure
7. 配置hadoop
7-1) 解压
下载地址：https://archive.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
tar xzvf hadoop-2.7.3.tar.gz -C /opt/
7-2) 创建存放数据的目录（必须事先创建好，否则会报错)
mkdir -p /opt/hadoop-2.7.3/data/full/tmp/ mkdir -p /opt/hadoop-2.7.3/data/full/tmp/dfs/name mkdir -p /opt/hadoop-2.7.3/data/full/tmp/dfs/data
7-3) 配置/opt/hadoop-2.7.3/etc/hadoop下面的配置文件
cd opt/hadoop-2.7.3/etc/hadoop #定位到配置文件目录
7-3-1) core-site.xml
<configuration> <!-- 指定HDFS中NameNode的地址 --> <property> <name>fs.defaultFS</name> <value>hdfs://127.0.0.1:9000</value> <description>hdfs://127.0.0.1:9000</description> </property> <!-- 指定hadoop运行时产生文件的存储目录 --> <property> <name>hadoop.tmp.dir</name> <value>/home/lenmom/workspace/software/hadoop-2.7.3/data/tmp</value> <description>是hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不配 置namenode和datanode的存放位置，默认就放在这个路径中</description> </property> <!--启用 webhdfs--> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> <description>启用 webhdfs</description> </property> <!--use hadoop native library--> <property> <name>hadoop.native.lib</name> <value>true</value> <description>Should native hadoop libraries, if present, be used.</description> </property> </configuration>
7-3-2) yarn-site.xml
<configuration> <property> <!-- reducer获取数据的方式 --> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <!-- 指定YARN的ResourceManager的地址 --> <name>yarn.resourcemanager.hostname</name> <value>palo253</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>palo253:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>palo253:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>palo253:8031</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>2.1</value> </property> </configuration>
7-3-3) slaves
palo252
palo253
palo254
7-3-4) mapred-site.xml
<configuration> <!-- 指定mr运行在yarn上 --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>palo252:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>palo252:19888</value> </property> </configuration>
7-3-5) hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> <description>不能大于datanode的数量,默认为3</description> </property> <!-- 设置secondname的端口 --> <property> <name>dfs.namenode.secondary.http-address</name> <value>palo253:50090</value> </property> <property> <name>dfs.data.dir</name> <value>file:/opt/hadoop-2.7.3/data/full/tmp/dfs/data</value> <description>用于确定将HDFS文件系统的数据保存在什么目录下,可以将这个参数设置为多个分区上目录，即可将HDFS建立在不同分区上。</description> </property> <property> <name>dfs.name.dir</name> <value>file:/opt/hadoop-2.7.3/data/full/tmp/dfs/name</value> <description>这个参数用于确定将HDFS文件系统的元信息保存在什么目录下,如果这个参数设置为多个目录，那么这些目录下都保存着元信息的多个备份.</description> </property> <!--设置 hadoop的代理用户--> <property> <name>hadoop.proxyuser.hadoop.hosts</name> <value>*</value> <description>配置成*的意义，表示任意节点使用 hadoop 集群的代理用户hadoop 都能访问 hdfs 集群</description> </property> <property> <name>hadoop.proxyuser.hadoop.groups</name> <value>*</value> <description>代理用户所属的组</description> </property> </configuration>
7-3-6) hadoop-env.sh
配置JAVA_HOME
1 # Licensed to the Apache Software Foundation (ASF) under one 2 # or more contributor license agreements. See the NOTICE file 3 # distributed with this work for additional information 4 # regarding copyright ownership. The ASF licenses this file 5 # to you under the Apache License, Version 2.0 (the 6 # "License"); you may not use this file except in compliance 7 # with the License. You may obtain a copy of the License at 8 # 9 # http://www.apache.org/licenses/LICENSE-2.0 10 # 11 # Unless required by applicable law or agreed to in writing, software 12 # distributed under the License is distributed on an "AS IS" BASIS, 13 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 # See the License for the specific language governing permissions and 15 # limitations under the License. 16 17 # Set Hadoop-specific environment variables here. 18 19 # The only required environment variable is JAVA_HOME. All others are 20 # optional. When running a distributed configuration it is best to 21 # set JAVA_HOME in this file, so that it is correctly defined on 22 # remote nodes. 23 24 # The java implementation to use. 25 #export JAVA_HOME=${JAVA_HOME} 26 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
8. 配置环境变量(每台机器都必须做)
vi /etc/profile
在文件尾部添加：
#####set jdk enviroment export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH ##### set hadoop_home enviroment export HADOOP_HOME=/opt/hadoop-2.7.3 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop export YARN_HOME=/home/lenmom/workspace/software/hadoop-2.7.3 export YARN_CONF_DIR=${YARN_HOME}/etc/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin ###enable hadoop native library export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
命令行终端执行 source /etc/profile，让配置的环境变量生效
source /etc/profile ####make the env variable to take effect right now.
9. 启动：
NameNode:(master 252)
#格式化namenode
hdfs namenode -format
#启动dfs
start-dfs.sh # (master 252)
#启动Yarn： yarn节点(253)
#注意：Namenode和ResourceManger如果不是同一台机器，
#不能在NameNode上启动 yarn，
#应该在ResouceManager所在的机器上启动yarn。
start-yarn.sh
#验证启动情况：
jps #查看java进程
http://namenode:50070/
10 Hadoop启动停止方式
1）各个服务组件逐一启动 分别启动hdfs组件： hadoop-daemon.sh start|stop namenode|datanode|secondarynamenode 启动yarn： yarn-daemon.sh start|stop resourcemanager|nodemanager 2）各个模块分开启动（配置ssh是前提）常用 start|stop-dfs.sh start|stop-yarn.sh 3）全部启动（不建议使用） start|stop-all.sh
4) 开启historyserver(任意节点启动即可)
mr-jobhistory-daemon.sh start|stop historyserver
reference:
