dremio RuleBasedEngineSelector 简单介绍

dremio 提供了一个空的默认RuleBasedEngineSelector 实现，主要实现基于规则的引擎选择

接口定义以及默认空实现

public interface RuleBasedEngineSelector {

 

  public String resolveAndUpdateEngine(ResourceSchedulingContext queryContext);

 

  /**

   * NO_OP implementation

   */

  public static final RuleBasedEngineSelector NO_OP = new RuleBasedEngineSelector() {

 

    @Override

    public String resolveAndUpdateEngine(ResourceSchedulingContext queryContext) {

      // 返回为空，不做操作

      return null;

    }

  };

 

}

使用此服务的地方

SabotNode（测试了不包含ui 的执行框架），ForemenWorkManager（foreman 管理的），AttemptManager（管理所有fragement）
主要使用的还是SabotNode 以及AttemptManager
AttemptManager 对于RuleBasedEngineSelector使用的处理(主要是将查询的规则附加到)

 

String ruleSetEngine = ruleBasedEngineSelector.resolveAndUpdateEngine(queryContext);

ResourceSchedulingProperties resourceSchedulingProperties = new ResourceSchedulingProperties();

resourceSchedulingProperties.setRoutingEngine(queryContext.getSession().getRoutingEngine());

resourceSchedulingProperties.setRuleSetEngine(ruleSetEngine);

final GroupResourceInformation groupResourceInformation =

  maestroService.getGroupResourceInformation(queryContext.getOptions(), resourceSchedulingProperties);

queryContext.setGroupResourceInformation(groupResourceInformation);

 

 

对于具体的使用是由BasicResourceAllocator进行具体的资源分配处理,但是很不好的是官方的实现没有直接使用此功能
反而使用的是基于队列的处理
参考代码

 

public ResourceSchedulingResult allocate(final ResourceSchedulingContext queryContext,

                                           final ResourceSchedulingProperties resourceSchedulingProperties,

                                           final ResourceSchedulingObserver resourceSchedulingObserver,

                                           final Consumer<ResourceSchedulingDecisionInfo> schedulingDecisionInfoConsumer) {

 

    final ResourceSchedulingDecisionInfo resourceSchedulingDecisionInfo = new ResourceSchedulingDecisionInfo();

    final QueueType queueType = getQueueNameFromSchedulingProperties(queryContext, resourceSchedulingProperties);

    resourceSchedulingDecisionInfo.setQueueName(queueType.name());

    resourceSchedulingDecisionInfo.setQueueId(queueType.name());

    resourceSchedulingDecisionInfo.setWorkloadClass(queryContext.getQueryContextInfo().getPriority().getWorkloadClass());

    schedulingDecisionInfoConsumer.accept(resourceSchedulingDecisionInfo);

 

    resourceSchedulingObserver.beginQueueWait();

    final Pointer<DistributedSemaphore.DistributedLease> lease = new Pointer();

    ListenableFuture<ResourceSet> futureAllocation = executorService.submit(() -> {

      lease.value = acquireQuerySemaphoreIfNecessary(queryContext, queueType);

 

       // update query limit based on the queueType

      final OptionManager options = queryContext.getOptions();

      // 此处基于了队列进行资源控制，但是没有利用上边的默认RuleBasedEngineSelector

      final boolean memoryControlEnabled = options.getOption(BasicResourceConstants.ENABLE_QUEUE_MEMORY_LIMIT);

      // TODO REFLECTION_SMALL, REFLECTION_LARGE was not there before - was it a bug???

      final long memoryLimit = (queueType == QueueType.SMALL || queueType == QueueType.REFLECTION_SMALL) ?

        options.getOption(BasicResourceConstants.SMALL_QUEUE_MEMORY_LIMIT):

        options.getOption(BasicResourceConstants.LARGE_QUEUE_MEMORY_LIMIT);

      long queryMaxAllocation = queryContext.getQueryContextInfo().getQueryMaxAllocation();

      if (memoryControlEnabled && memoryLimit > 0) {

        queryMaxAllocation = Math.min(memoryLimit, queryMaxAllocation);

      }

      final UserBitShared.QueryId queryId = queryContext.getQueryId();

      final long queryMaxAllocationFinal = queryMaxAllocation;

 

      final ResourceSet resourceSet = new BasicResourceSet(

        queryId,

        lease.value,

        queryMaxAllocationFinal,

        queueType.name());

 

      return resourceSet;

    });

    Futures.addCallback(futureAllocation, new FutureCallback<ResourceSet>() {

      @Override

      public void onSuccess(@Nullable ResourceSet resourceSet) {

        // don't need to do anything additional

      }

 

      @Override

      public void onFailure(Throwable throwable) {

        // need to close lease

        releaseLease(lease.value);

      }

    }, executorService);

 

    final ResourceSchedulingResult resourceSchedulingResult = new ResourceSchedulingResult(

      resourceSchedulingDecisionInfo,

      futureAllocation

    );

    return resourceSchedulingResult;

  }

说明

RuleBasedEngineSelector 功能设计上主要是一个接口定义，对于资源分配需要依赖的可以直接使用，但是目前开源版本的比较弱

参考资料

https://docs.dremio.com/software/advanced-administration/jobs/