dremio 的InformationSchemaCatalog 服务一

InformationSchemaCatalog 服务实现了模仿information_schema的能力，让我们可以更好的在bi 工具以及其他系统使用

接口定义

/**

 * Facet of the catalog service that provides metadata with an information_schema like API.

 * <p>

 * The metadata provided may not satisfy the configured metadata policy requirement of various sources; the reads

 * are directly from the KVStore without an inline refresh of the metadata.

 */

interface InformationSchemaCatalog {

 

  /**

   * List catalogs that satisfy the given search query.

   *

   * @param searchQuery        search query, may be null

   * @return iterator of catalogs

   */

  Iterator<Catalog> listCatalogs(SearchQuery searchQuery);

 

  /**

   * List schemata that satisfy the given search query.

   *

   * @param searchQuery        search query, may be null

   * @return iterator of schemata

   */

  Iterator<Schema> listSchemata(SearchQuery searchQuery);

 

  /**

   * List tables (datasets) that satisfy the given search query.

   *

   * @param searchQuery        search query, may be null

   * @return iterator of tables

   */

  Iterator<Table> listTables(SearchQuery searchQuery);

 

  /**

   * List views that satisfy the given search query.

   *

   * @param searchQuery        search query, may be null

   * @return iterator of views

   */

  Iterator<View> listViews(SearchQuery searchQuery);

 

  /**

   * List columns of all datasets that satisfy the given search query.

   *

   * @param searchQuery        search query, may be null

   * @return iterator of columns

   */

  Iterator<TableSchema> listTableSchemata(SearchQuery searchQuery);

 

}

子类

 

 

从上图可以看出InformationSchemaCatalog 的直接实现是没有包含PrivilegeCatalog，catalog 的子类需要实现此接口，具体的实现InformationSchemaCatalogImpl
核心依赖NamespaceService（此服务比较重要，dremio 利用了key,value 存储解决数据状态管理）
比如数据表获取的

 

 @Override

  public Iterator<Table> listTables(SearchQuery searchQuery) {

    // 基于NamespaceService进行处理

    final Iterable<Map.Entry<NamespaceKey, NameSpaceContainer>> searchResults =

      userNamespace.find(getCondition(searchQuery));

 

    return StreamSupport.stream(searchResults.spliterator(), false)

      .filter(IS_NOT_INTERNAL)

      .filter(IS_DATASET)

      .map(input -> {

        final String sourceName = input.getKey().getRoot();

 

        final TableType tableType;

        if (input.getValue().getDataset().getType() == DatasetType.VIRTUAL_DATASET) {

          tableType = TableType.VIEW;

        } else if ("sys".equals(sourceName) || "INFORMATION_SCHEMA".equals(sourceName)) {

         // 包含了sys 以及INFORMATION_SCHEMA 的处理，sys 数据dremio 内部的，INFORMATION_SCHEMA 属于标准的INFORMATION_SCHEMA

          tableType = TableType.SYSTEM_TABLE;

        } else {

          tableType = TableType.TABLE;

        }

 

        return Table.newBuilder()

          .setCatalogName(DEFAULT_CATALOG_NAME)

          .setSchemaName(input.getKey().getParent().toUnescapedString())

          .setTableName(input.getKey().getName())

          .setTableType(tableType)

          .build();

      })

      .iterator();

  }

说明

InformationSchemaCatalog 的实现，完善了dremio 对于bi 以及jdbc 周边工具的集成，实现上不难，主要是利用了NamespaceService 服务

参考资料

sabot/kernel/src/main/java/com/dremio/exec/catalog/InformationSchemaCatalog.java
https://docs.dremio.com/software/sql-reference/information-schema/tables/