Yarn 集群环境搭建

环境准备

确保主机搭建 HDFS 运行环境

步骤一：修改 mapred-site.xml 配置文件

复制代码

 [root@node-01 ~]# cd /root/apps/hadoop-3.2.1/etc/hadoop/
 [root@node-01 hadoop]# vim mapred-site.xml
 <configuration>
     	<!-- 配置MapReduce程序运行模式 为 yarn（不配置默认为 local 模式) -->
        <property>
                  <name>mapreduce.framework.name</name>
                  <value>yarn</value>
         </property>
     
		<!-- 设置 hadoop 路径 -->
		<property>
        	<name>mapreduce.application.classpath</name>
        	<value>/root/apps/hadoop-3.2.1/etc/hadoop:/root/apps/hadoop-3.2.1/share/hadoop/common/lib/*:/root/apps/hadoop-3.2.1/share/hadoop/common/*:/root/apps/hadoop-3.2.1/share/hadoop/hdfs:/root/apps/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/root/apps/hadoop-3.2.1/share/hadoop/hdfs/*:/root/apps/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/root/apps/hadoop-3.2.1/share/hadoop/mapreduce/*:/root/apps/hadoop-3.2.1/share/hadoop/yarn:/root/apps/hadoop-3.2.1/share/hadoop/yarn/lib/*:/root/apps/hadoop-3.2.1/share/hadoop/yarn/*</value>
		</property>

</configuration>

步骤二：修改yarn-env.sh 配置文件

复制代码

[root@node-01 ~]# cd /root/apps/hadoop-3.2.1/etc/hadoop
[root@node-01 hadoop]# echo 'export JAVA_HOME=${JAVA_HOME}' >> yarn-env.sh

步骤三：修改 yarn-site.xml 配置文件

复制代码

[root@node-01 ~]# cd /root/apps/hadoop-3.2.1/etc/hadoop/
[root@node-01 hadoop]# vim yarn-site.xml
<configuration>
<!-- 配置 NodeManager上运行的附属服务（指定 MapReduce 中 reduce 读取数据方式） -->
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
    
<property>
        <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
</property>

<property>
          <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
          <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>node-01:8025</value>
</property>

<property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>node-01:8030</value>
</property>

<property>
        <name>yarn.resourcemanager.address</name>
         <value>node-01:8050</value>
</property>

    
<!-- 配置自动检测硬件（默认关闭) -->
<property>
        <name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
        <value>true</value>
</property>
    
 <!-- 配置 nodemanager 启动要求的最低配置-->
<property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
</property>
<property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
</property>

</configuration>

步骤四：scp 这个 yarn-site.xml 到其他节点

复制代码

[root@node-01 ~]# cd /root/apps/hadoop-3.2.1/etc/hadoop/
[root@node-01 ~]# scp mapred-site.xml node-02:$PWD
[root@node-01 ~]# scp mapred-site.xml node-03:$PWD
[root@node-01 ~]# scp yarn-env.sh node-02:$PWD
[root@node-01 ~]# scp yarn-env.sh node-03:$PWD
[root@node-01 ~]# scp yarn-site.xml node-02:$PWD
[root@node-01 ~]# scp yarn-site.xml node-03:$PWD

步骤五：启动 yarn 集群

注：先启动 HDFS HA 集群

复制代码

[root@node-01 ~]# start-yarn.sh

stop-yarn.sh ：停止 yarn 集群

步骤六：用 jps 检查 yarn 的进程

复制代码

[root@node-01 ~]# jps
6128 DFSZKFailoverController
5939 JournalNode
6468 NodeManager
5544 NameNode
5289 QuorumPeerMain
6314 ResourceManager
5691 DataNode

[root@node-02 ~]# jps
3201 DFSZKFailoverController
2962 DataNode
3284 NodeManager
3061 JournalNode
2877 NameNode
2751 QuorumPeerMain

[root@node-03 ~]# jps
2323 QuorumPeerMain
2444 DataNode
2652 NodeManager
2543 JournalNode