大叔经验分享（21）yarn中查看每个应用实时占用的内存和cpu资源

在yarn中的application详情页面

http://resourcemanager/cluster/app/$applicationId

或者通过application命令

yarn application -status $applicationId

只能看到应用启动以来占用的资源*时间统计，比如：

Aggregate Resource Allocation : 3962853 MB-seconds, 1466 vcore-seconds

到处都找不到这个应用当前实时的资源占用情况，比如当前占用了多少内存多少核，跟进yarn代码发现其实是有这个统计的：

org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport

  public static ApplicationResourceUsageReport newInstance(
      int numUsedContainers, int numReservedContainers, Resource usedResources,
      Resource reservedResources, Resource neededResources, long memorySeconds,
      long vcoreSeconds) {
    ApplicationResourceUsageReport report =
        Records.newRecord(ApplicationResourceUsageReport.class);
    report.setNumUsedContainers(numUsedContainers);
    report.setNumReservedContainers(numReservedContainers);
    report.setUsedResources(usedResources);
    report.setReservedResources(reservedResources);
    report.setNeededResources(neededResources);
    report.setMemorySeconds(memorySeconds);
    report.setVcoreSeconds(vcoreSeconds);
    return report;
  }

其中usedResources就是当前的实时占用资源情况，包括内存和cpu，这个统计是在YarnScheduler的接口中返回：

org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

  /**
   * Get a resource usage report from a given app attempt ID.
   * @param appAttemptId the id of the application attempt
   * @return resource usage report for this given attempt
   */
  @LimitedPrivate("yarn")
  @Evolving
  ApplicationResourceUsageReport getAppResourceUsageReport(
      ApplicationAttemptId appAttemptId);

getAppResourceUsageReport方法被RMAppAttemptImpl.getApplicationResourceUsageReport调用：

org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl

  @Override
  public ApplicationResourceUsageReport getApplicationResourceUsageReport() {
    this.readLock.lock();
    try {
      ApplicationResourceUsageReport report =
          scheduler.getAppResourceUsageReport(this.getAppAttemptId());
      if (report == null) {
        report = RMServerUtils.DUMMY_APPLICATION_RESOURCE_USAGE_REPORT;
      }
      AggregateAppResourceUsage resUsage =
          this.attemptMetrics.getAggregateAppResourceUsage();
      report.setMemorySeconds(resUsage.getMemorySeconds());
      report.setVcoreSeconds(resUsage.getVcoreSeconds());
      return report;
    } finally {
      this.readLock.unlock();
    }
  }

RMAppAttemptImpl.getApplicationResourceUsageReport被两个地方调用：

第一个调用

org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl

  public ApplicationReport createAndGetApplicationReport(String clientUserName,
      boolean allowAccess) {
...
          appUsageReport = currentAttempt.getApplicationResourceUsageReport();
...

RMAppImpl.createAndGetApplicationReport会被ClientRMService.getApplications和ClientRMService.getApplicationReport调用，这两个方法分别对应命令

yarn application -list
yarn application -status $applicationId

这两个地方展示信息的时候都没展示usedResources，可能作者觉得这个实时资源占用统计没那么重要。

详见：
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService

第二个调用

org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo

  public AppInfo(RMApp app, Boolean hasAccess, String schemePrefix) {
...
          ApplicationResourceUsageReport resourceReport = attempt
              .getApplicationResourceUsageReport();
          if (resourceReport != null) {
            Resource usedResources = resourceReport.getUsedResources();
            allocatedMB = usedResources.getMemory();
            allocatedVCores = usedResources.getVirtualCores();
            runningContainers = resourceReport.getNumUsedContainers();
          }
...

这个构造函数会在RMWebServices.getApp和RMWebServices.getApps时被调用，这是个service接口，对应url分别为：

http://resourcemanager/ws/v1/cluster/apps/$applicationId
http://resourcemanager/ws/v1/cluster/apps?state=RUNNING

这两个接口的返回值中有实时资源占用情况如下：

<allocatedMB>56320</allocatedMB>
<allocatedVCores>21</allocatedVCores>

分别对应实时内存占用和实时CPU占用；

详见：
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices

 

如果你发现spark应用内存的占用比你分配的要多，可以参考这里：https://www.cnblogs.com/barneywill/p/10102353.html