Hadoop Yarn

Yarn Architecture

Yarn Scheduler

看起来不同的version，默认的scheduler不同。题主所试hadoop2.6.0默认的是CapacityScheduler，cdh5.1.0默认的是FairScheduler。

如何查看当前集群所配置的scheduler

1.在 Resource Manager web UI scheduler里面查看。e.g.http://vm-slaver1:8088/cluster/scheduler

2.yarn-site.xml 里面查看 yarn.resourcemanager.scheduler.class 配置，通常如果没有配置，去看对应版本的yarn-default.xml

CapacityScheduler

在配置文件目录下面etc/hadoop，找到或新建一个capacity-scheduler.xml. 然后配置scheduler相关参数。

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

example: capacity-scheduler.xml

<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>q0</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

   <property>
     <name>yarn.scheduler.capacity.root.q0.queues</name>
     <value>q01,q02</value>
   </property>      

  <property>
    <name>yarn.scheduler.capacity.root.q0.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q01.capacity</name>
    <value>60</value>
    <description>Default queue target capacity.</description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q02.capacity</name>
    <value>40</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q01.maximum-capacity</name>
    <value>80</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

<property>
    <name>yarn.scheduler.capacity.root.q0.q02.maximum-capacity</name>
    <value>80</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.q01.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

 <property>
    <name>yarn.scheduler.capacity.root.q0.q02.state</name>
    <value>STOPPED</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.q0.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers. 
      Typically this should be set to number of nodes in the cluster, By default is setting 
      approximately number of nodes in one rack which is 40.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

</configuration>

Tips:

1. The sum of capacities for all queues, at each level, must be equal to 100. e.g. q0.q01+q0.q02=100

 

FairScheduler

默认fair scheduler的配置文件为fair-scheduler.xml，可以通过yarn-site.xml 里面的yarn.scheduler.fair.allocation.file指定。

 

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

 

examples: fair-scheduler-allocations.xml

<?xml version="1.0"?>
<allocations>
  <queue name="q0">
    <minResources>10000 mb,0vcores</minResources>
    <maxResources>90000 mb,1vcores</maxResources>
    <maxRunningApps>50</maxRunningApps>
    <maxAMShare>0.1</maxAMShare>
    <weight>2.0</weight>
    <schedulingPolicy>fair</schedulingPolicy>
    <queue name="q01">
      <aclSubmitApps>hadoop</aclSubmitApps>
      <minResources>5000 mb,0vcores</minResources>
    </queue>
  </queue>

  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>

  <queue name="q1" type="parent">
  <weight>3.0</weight>
  </queue>
  
  <user name="hadoop">
    <maxRunningApps>30</maxRunningApps>
  </user>
  <userMaxAppsDefault>5</userMaxAppsDefault>
  
  <queuePlacementPolicy>
    <rule name="specified" />
    <rule name="primaryGroup" create="false" />
    <rule name="nestedUserQueue">
        <rule name="secondaryGroupExistingQueue" create="false" />
    </rule>
    <rule name="default" queue="q0"/>
  </queuePlacementPolicy>
</allocations>

在scheduler UI上，会是这样：

Tips:

在 ApplicationSubmissionContext setQueue时，name必须是leaf queue，且带除root外的根路径名。e.g. q0.q01 or root.q0.q1

（注：但是后来在spark research的时候，发现spark里面只要直接设置叶子节点名即可，sparkConf.set("spark.yarn.queue","q01");

HelloWorld for Yarn Application