Yarn Node Labels
Yarn Node Labels + Capacity-Scheduler
在yarn-site.xml中开启capacity-schedule
yarn-site.xml <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property>
配置capacity-scheduler.xml文件
<configuration> <property> <name>yarn.scheduler.capacity.root.queues</name> <value>default,area0,area1,area2</value> <description> The queues at the this level (root is the root queue). </description> </property> <property> <name>yarn.scheduler.capacity.root.default.capacity</name> <value>25</value> </property> <property> <name>yarn.scheduler.capacity.root.area0.capacity</name> <value>25</value> </property> <property> <name>yarn.scheduler.capacity.root.area1.capacity</name> <value>25</value> </property> <property> <name>yarn.scheduler.capacity.root.area2.capacity</name> <value>25</value> </property> <property> <name>yarn.scheduler.capacity.root.default.maximum-capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.area0.maximum-capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.area1.maximum-capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.area2.maximum-capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels</name> <value>*</value> </property> <property> <name>yarn.scheduler.capacity.root.area0.accessible-node-labels</name> <value>area0</value> </property> <property> <name>yarn.scheduler.capacity.root.area1.accessible-node-labels</name> <value>area1</value> </property> <property> <name>yarn.scheduler.capacity.root.area2.accessible-node-labels</name> <value>area2</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.area0.capacity</name> <value>33</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.area1.capacity</name> <value>33</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.area2.capacity</name> <value>34</value> </property> <property> <name>yarn.scheduler.capacity.root.area0.accessible-node-labels.area0.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.area1.accessible-node-labels.area1.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.area2.accessible-node-labels.area2.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.default-node-label-expression</name> <value> ,area0,area1,area2</value> </property> <!--value为空,则root.default队列只能在没有label的node上执行,而我环境环境中不存在没有label的node,所以提交到root.default的任务卡死--> <property> <name>yarn.scheduler.capacity.root.default.default-node-label-expression</name> <value> </value> </property> <property> <name>yarn.scheduler.capacity.root.area0.default-node-label-expression</name> <value>area0</value> </property> <property> <name>yarn.scheduler.capacity.root.area1.default-node-label-expression</name> <value>area1</value> </property> <property> <name>yarn.scheduler.capacity.root.area2.default-node-label-expression</name> <value>area2</value> </property> </configuration>
添加Yarn Node Labels
yarn rmadmin -addToClusterNodeLabels area0,area1,area2
#节点添加标签
yarn rmadmin -replaceLabelsOnNode Host1:45454,area0
yarn rmadmin -replaceLabelsOnNode Host2:45454,area1
yarn rmadmin -replaceLabelsOnNode Host3:45454,area2
#查看标签
yarn node -status Host1:45454
#也可以通过Yarn管理页面查看Node Label
配置Yarn Node Labels存储目录,这样不用每次重启服务后重新配置Yarn Node Labels。
在yarn-site.xml中添加下列信息
<property>
<name>yarn.node-labels.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:45454</value>
</property>
<property>
<name>yarn.node-labels.manager-class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager</value>
</property>
<property>
<name>yarn.node-labels.fs-store.root-dir</name>
<value>hdfs://Host0:8020/yarn/node-labels</value>
<description>标签数据在HDFS上的存储位置</description>
</property>
http://www.cnblogs.com/suanec/
友链:marsggbo
声援博主:如果您觉得文章对您有帮助,可以点击文章右下角【推荐】一下。您的鼓励是博主的最大动力!
点个关注吧~