dremio FragmentExecutor 的执行顺序简单说明

dremio 在执行计划物理计划转换之后，对于执行计划会包含不同的fragment，fragment 会组成一颗树，包含了PlanFragmentMajor以及PlanFragmentMinor
对于组成的树之后dremio 就需要调度执行了（里边会包含资源分配，优先级，运算操作,大致处理可以参考drill 查询执行介绍）

FragmentExecutors 中的启动Fragments

参考如下，通过QueryStarterImpl 包装进行处理

 

 public void startFragments(final InitializeFragments fragments, final FragmentExecutorBuilder builder,

                             final StreamObserver<Empty> sender, final NodeEndpoint identity) {

    final SchedulingInfo schedulingInfo = fragments.hasSchedulingInfo() ? fragments.getSchedulingInfo() : null;

    QueryStarterImpl queryStarter = new QueryStarterImpl(fragments, builder, sender, identity, schedulingInfo);

    // 通过FragmentExecutorBuilder 构建FragmentExecutor，内部使用了QueriesClerk 支持工作负载以及查询级别的分配处理，包装完成

   // 之后还是调用的QueryStarterImpl 内部的buildAndStartQuery

    builder.buildAndStartQuery(queryStarter.getFirstFragment(), schedulingInfo, queryStarter);

  }

执行顺序处理

QueryStarterImpl 中的buildAndStartQuery 

....

public void buildAndStartQuery(final QueryTicket queryTicket) {

      QueryId queryId = queryTicket.getQueryId();

 

      /**

       * To avoid race conditions between creation and deletion of phase/fragment tickets,

       * build all the fragments first (creates the tickets) and then, start the fragments (can

       * delete tickets).

       */

      List<FragmentExecutor> fragmentExecutors = new ArrayList<>();

      UserRpcException userRpcException = null;

      Set<FragmentHandle> fragmentHandlesForQuery = Sets.newHashSet();

      try {

        if (!maestroProxy.tryStartQuery(queryId, queryTicket, initializeFragments.getQuerySentTime())) {

          boolean isDuplicateStart = maestroProxy.isQueryStarted(queryId);

          if (isDuplicateStart) {

            // duplicate op, do nothing.

            return;

          } else {

            throw new IllegalStateException("query already cancelled");

          }

        }

 

        Map<Integer, Integer> priorityToWeightMap = buildPriorityToWeightMap();

 

        for (PlanFragmentFull fragment : fullFragments) {

          // 构建FragmentExecutor

          FragmentExecutor fe = buildFragment(queryTicket, fragment, priorityToWeightMap.getOrDefault(fragment.getMajor().getFragmentExecWeight(), 1), schedulingInfo);

          fragmentHandlesForQuery.add(fe.getHandle());

          fragmentExecutors.add(fe);

        }

      } catch (UserRpcException e) {

        userRpcException = e;

      } catch (Exception e) {

        userRpcException = new UserRpcException(NodeEndpoint.getDefaultInstance(), "Remote message leaked.", e);

      } finally {

        if (fragmentHandlesForQuery.size() > 0) {

          maestroProxy.initFragmentHandlesForQuery(queryId, fragmentHandlesForQuery);

        }

        // highest weight first with leaf fragments first bottom up

       // 进行排序处理，然后执行FragmentExecutor，实际的处理是可以参考drill 的查询执行介绍的

        fragmentExecutors.sort(weightBasedComparator());

        for (FragmentExecutor fe : fragmentExecutors) {

          startFragment(fe);

        }

        queryTicket.release();

 

        if (userRpcException == null) {

          sender.onNext(Empty.getDefaultInstance());

          sender.onCompleted();

        } else {

          sender.onError(userRpcException);

        }

      }

 

      // if there was a cancel while the start was in-progress, clean up.

      if (maestroProxy.isQueryCancelled(queryId)) {

        cancelFragments(queryId, builder.getClerk());

      }

    }

buildFragment 构建FragmentExecutor

 private FragmentExecutor buildFragment(final QueryTicket queryTicket, final PlanFragmentFull fragment,

      final int schedulingWeight, final SchedulingInfo schedulingInfo) throws UserRpcException {

 

      if (fragment.getMajor().getFragmentExecWeight() <= 0) {

        logger.info("Received remote fragment start instruction for {}", QueryIdHelper.getQueryIdentifier(fragment.getHandle()));

      } else {

        logger.info("Received remote fragment start instruction for {} with assigned weight {} and scheduling weight {}",

          QueryIdHelper.getQueryIdentifier(fragment.getHandle()), fragment.getMajor().getFragmentExecWeight(), schedulingWeight);

      }

 

      try {

        final EventProvider eventProvider = getEventProvider(fragment.getHandle());

        return builder.build(queryTicket, fragment, schedulingWeight, useMemoryArbiter ? memoryArbiter : null, eventProvider, schedulingInfo, fragmentReader);

      } catch (final Exception e) {

        throw new UserRpcException(identity, "Failure while trying to start remote fragment", e);

      } catch (final OutOfMemoryError t) {

        if (t.getMessage().startsWith("Direct buffer")) {

          throw new UserRpcException(identity, "Out of direct memory while trying to start remote fragment", t);

        } else {

          throw t;

        }

      }

    }

权重比较

 // 通过PlanFragmentFull 获取执行的权重，此生成是根据，dremio 的简单并行化处理的（SimpleParallelizer，内部会通过资源的处理进行评估，后边详细介绍）

private Map<Integer, Integer> buildPriorityToWeightMap() {

      Map<Integer, Integer> priorityToWeightMap = new HashMap<>();

      if (fullFragments.get(0).getMajor().getFragmentExecWeight() <= 0) {

        return priorityToWeightMap;

      }

 

      for(PlanFragmentFull fragment : fullFragments) {

        int fragmentWeight = fragment.getMajor().getFragmentExecWeight();

        priorityToWeightMap.put(fragmentWeight, Math.min(fragmentWeight, QueryTicket.MAX_EXPECTED_SIZE));

      }

      return priorityToWeightMap;

    }

   // 进行比较的，核心是处理root ，leaf 已经一些FragmentExecutor 的顺序，然后进行执行

    private Comparator<FragmentExecutor> weightBasedComparator() {

      return (e1, e2) -> {

        if (e2.getFragmentWeight() < e1.getFragmentWeight()) {

          // higher priority first

          return -1;

        }

        if (e2.getFragmentWeight() == e1.getFragmentWeight()) {

          // when priorities are equal, order the fragments based on leaf first and then descending order

          // of major fragment number. This ensures fragments are started in order of dependency

          if (e2.isLeafFragment() == e1.isLeafFragment()) {

            return Integer.compare(e2.getHandle().getMajorFragmentId(), e1.getHandle().getMajorFragmentId());

          } else {

            return e1.isLeafFragment() ? -1 : 1;

          }

        }

        return 1;

      };

    }

说明

实际上通过阅读drill 的查询执行介绍可以可以看出来顺序，但是通过源码的阅读可以更好的学习了解内部处理

参考资料

sabot/kernel/src/main/java/com/dremio/sabot/exec/fragment/FragmentExecutor.java
sabot/kernel/src/main/java/com/dremio/sabot/exec/FragmentExecutors.java
sabot/kernel/src/main/java/com/dremio/exec/planner/fragment/PlanFragmentFull.java
sabot/kernel/src/main/java/com/dremio/sabot/exec/WorkloadTicket.java
sabot/kernel/src/main/java/com/dremio/sabot/exec/fragment/FragmentExecutorBuilder.java
sabot/kernel/src/main/java/com/dremio/exec/planner/fragment/SimpleParallelizer.java
https://drill.apache.org/docs/drill-query-execution/
https://www.cnblogs.com/rongfengliang/p/17008637.html
https://www.cnblogs.com/rongfengliang/p/17008654.html