dremio ViewCreatorFactory 简单说明

ViewCreatorFactory 是dremio 中专门管理view 的，目前主要是对于space 以及home 中的处理，当然对于source 也是可以支持的（支持版本类型的，比如nessie，原始分布式文件系统的实际也是可以支持的，但是不太符合dremio的玩法，以及限定死了）

调用部分

实际上通过标准的sqlhandler，具体实现就是CreateViewHandler

• 参考代码

CreateViewHandler.java

public List<SimpleCommandResult> toResult(String sql, SqlNode sqlNode) throws Exception {

    SqlCreateView createView = SqlNodeUtil.unwrap(sqlNode, SqlCreateView.class);

    ParserUtil.validateParsedViewQuery(createView.getQuery());

    final NamespaceKey resolvedViewPath = catalog.resolveSingle(createView.getPath());

    Preconditions.checkState(resolvedViewPath.size() > 1, "View path "+ resolvedViewPath + " is not valid ");

    String formattedViewSql = getViewSql(createView, sql);

    VersionContext statementSourceVersion =

      ReferenceTypeUtils.map(createView.getRefType(), createView.getRefValue(), null);

    final VersionContext sessionVersion = config.getContext().getSession().getSessionVersionForSource(resolvedViewPath.getRoot());

    ConvertedRelNode convertedRelNode = validateTablesAndVersionContext(createView.getQuery(), resolvedViewPath, statementSourceVersion.orElse(sessionVersion));

    catalog.validatePrivilege(resolvedViewPath, SqlGrant.Privilege.ALTER);

    if (isVersioned(resolvedViewPath)) {

      return createVersionedView(createView, formattedViewSql, convertedRelNode, resolvedViewPath);

    } else {

      return createView(createView, formattedViewSql, convertedRelNode, resolvedViewPath);

    }

  }

createView参考处理

• 代码

CatalogImpl.java

 public void createView(final NamespaceKey key, View view, ViewOptions viewOptions, NamespaceAttribute... attributes) throws IOException {

    // 会判断不同的类型，比如source，space，home

    switch (getType(key, true)) {

      case SOURCE:

        asMutable(key, "does not support create view")

          .createOrUpdateView(key, options.getSchemaConfig(), view, viewOptions);

        break;

      case SPACE:

      case HOME:

        if (view.hasDeclaredFieldNames()) {

          throw UserException.unsupportedError()

              .message("Dremio doesn't support field aliases defined in view creation.")

              .buildSilently();

        }

        // 对于space 以及home 都使用了viewCreatorFactory 进行view 的操作，具体的实现是DACViewCreatorFactory

        viewCreatorFactory.get(userName)

          .createView(key.getPathComponents(), view.getSql(), view.getWorkspaceSchemaPath(), false, attributes);

        break;

      default:

        throw UserException.unsupportedError()

          .message("Cannot create view in %s.", key)

          .build(logger);

    }

  }

DACViewCreatorFactory的处理

• 初始化

DACDaemonModule 模块中
参考代码

registry.bind(

      ViewCreatorFactory.class,

      new DACViewCreatorFactory(

        registry.provider(InitializerRegistry.class),

        registry.provider(LegacyKVStoreProvider.class),

        registry.provider(JobsService.class),

        registry.provider(NamespaceService.Factory.class),

        registry.provider(CatalogService.class),

        registry.provider(ContextService.class),

        () -> bootstrap.getAllocator()

      )

);

• createView 处理

DACViewCreatorFactory 实现了ViewCreatorFactory 的接口定义，我只简单说明下createView 的处理

public void createView(List<String> path, String sql, List<String> sqlContext, boolean isVersionedSource, NamespaceAttribute... attributes) {

      SecurityContext securityContext = getSecurityContext();

      QueryExecutor executor = new QueryExecutor(jobsService, null, securityContext);

      DatasetTool tool = newDatasetTool(securityContext, executor);

 

      try {

        DatasetVersion version = DatasetVersion.newVersion();

        DatasetPath datasetPath = new DatasetPath(path);

       // DatasetTool 会进行view dataset 的创建，只是默认title为空，同时加载预览数据

        InitialPreviewResponse response = tool.newUntitled(allocator, new FromSQL(sql), version, sqlContext, null,true, 0, true);

        DatasetPath tmpPath = new DatasetPath(response.getDataset().getFullPath());

        VirtualDatasetUI vds = datasetService.getVersion(tmpPath, response.getDataset().getDatasetVersion(), isVersionedSource);

        // TODO - Remove dependency on a Resource file.

       // 此处使用了一个DatasetVersionResource，不太好，上边的todo 也说明了

        newDatasetVersionResource(securityContext, tool, version, tmpPath, allocator).save(vds, datasetPath, null, null, isVersionedSource, attributes);

      } catch (Exception e) {

        throw Throwables.propagate(e);

      }

}

说明

dremio 推荐的玩法是基于view 的vds 处理，使用了ViewCreatorFactory，核心是space 以及home 类型的，当然对于支持多版本的source 也是支持的（nessie ），以上是一个简单说明，详细的可以参考个github 源码（比如CreateViewHandler 执行的）

参考资料

sabot/kernel/src/main/java/com/dremio/exec/catalog/ViewCreatorFactory.java
dac/backend/src/main/java/com/dremio/dac/service/datasets/DACViewCreatorFactory.java
sabot/kernel/src/main/java/com/dremio/exec/catalog/CatalogImpl.java
sabot/kernel/src/main/java/com/dremio/exec/planner/sql/parser/SqlCreateView.java
sabot/kernel/src/main/java/com/dremio/exec/server/SabotContext.java
sabot/kernel/src/main/java/com/dremio/exec/planner/sql/handlers/direct/CreateViewHandler.java
sabot/kernel/src/main/java/com/dremio/exec/planner/sql/handlers/commands/CommandCreator.java
sabot/kernel/src/main/java/com/dremio/exec/work/foreman/AttemptManager.java
dac/backend/src/main/java/com/dremio/dac/explore/QueryExecutor.java
dac/backend/src/main/java/com/dremio/dac/explore/DatasetTool.java
dac/backend/src/main/java/com/dremio/dac/service/datasets/DatasetVersionMutator.java