Yarn Scheduler Load Simulator (SLS)-调度算法选择源码解析
如何为SLS选择调度算法。这里谈下本人的理解
sls通过slsrun.sh脚本来运行,其中要加入配置文件sls-runner.xml。在该配置文件中,有关于调度算法的配置:
</property>
<property>
<name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler</name>
<value>org.apache.hadoop.yarn.sls.scheduler.FifoSchedulerMetrics</value>
</property>
<property>
<name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</name>
<value>org.apache.hadoop.yarn.sls.scheduler.FairSchedulerMetrics</value>
</property>
<property>
<name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</name>
<value>org.apache.hadoop.yarn.sls.scheduler.CapacitySchedulerMetrics</value>
</property>
这是配置了三种调度算法的具体实现在哪。
但没有选择用哪种算法。
算法的选择在yarn配置时就确定好了,在yarn-site.xml文件中进行配置。
Property===>yarn.resourcemanager.scheduler.class
Value=====>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
在sls源码中也可以看出:
/hadoop-trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
private void startRM() throws ClassNotFoundException, YarnException {
Configuration rmConf = new YarnConfiguration(getConf());
String schedulerClass = rmConf.get(YarnConfiguration.RM_SCHEDULER);
if (Class.forName(schedulerClass) == CapacityScheduler.class) {
rmConf.set(YarnConfiguration.RM_SCHEDULER,
SLSCapacityScheduler.class.getName());
rmConf.setBoolean(YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
rmConf.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
ProportionalCapacityPreemptionPolicy.class.getName());
} else if (Class.forName(schedulerClass) == FairScheduler.class) {
rmConf.set(YarnConfiguration.RM_SCHEDULER,
SLSFairScheduler.class.getName());
} else if (Class.forName(schedulerClass) == FifoScheduler.class) {
// TODO add support for FifoScheduler
throw new YarnException("Fifo Scheduler is not supported yet.");
}
rmConf.set(SLSConfiguration.METRICS_OUTPUT_DIR, metricsOutputDir);
final SLSRunner se = this;
rm = new ResourceManager() {
@Override
protected ApplicationMasterLauncher createAMLauncher() {
return new MockAMLauncher(se, this.rmContext, amMap);
}
};
// Across runs of parametrized tests, the JvmMetrics objects is retained,
// but is not registered correctly
JvmMetrics jvmMetrics = JvmMetrics.initSingleton("ResourceManager", null);
jvmMetrics.registerIfNeeded();
// Init and start the actual ResourceManager
rm.init(rmConf);
rm.start();
}
在start RM开始时,首先从YARN configuration(我猜想是yarn-site.xml)中读取Scheduler的类别,也就是三种中的一种,然后与sls的sls-runner.xml的配置值相比,找到name相对应的配置值。到此,选择好了算法。
posted on 2018-06-23 15:03 sichenzhao 阅读(204) 评论(0) 编辑 收藏 举报