dremio CatalogService 服务简单说明
CatalogService 在dremio 中还是比较重要的，包含了source 以及storage 插件的通信处理，ManagedStoragePlugin 实际上是一个存储扩展的
包装，支持插件的状态管理，PluginsManager 主要进行存储扩展的管理，如果看过我以前写过的jdbc arp加载处理应该也能知道，插件的创建
以及管理就是使用了此服务，结合下图应该也能看到 CatalogService 提供的能力，同时CatalogService 服务还包含了创建Catalog的能力（对于使用
还是很有用的）
简单说明
- 接口定义
- 参考实现
服务的初始化
以下代码来自官方的测试用例 sabot/kernel/src/test/java/com/dremio/exec/catalog/TestDatasetCatalogServiceImpl.java
还是比较复杂的，基本依赖了不少dremio 的核心模块（kvstore，scheduler，一些内置扩展）
catalogService = new CatalogServiceImpl(
() -> sabotContext,
() -> new LocalSchedulerService(1),
() -> new SystemTablePluginConfigProvider(),
null,
() -> fabricService,
() -> ConnectionReader.of(sabotContext.getClasspathScan(), sabotConfig),
() -> allocator,
() -> storeProvider,
() -> datasetListingService,
() -> optionManager,
() -> broadcaster,
dremioConfig,
EnumSet.allOf(ClusterCoordinator.Role.class)
);
catalogService.start();
服务启动(包含的服务还是不少的，核心主要还是对于插件的管理，以及fabric rpc 服务，还有就是基于运行节点的状态创建一些内置的插件)
注意从源码可以看出主要在master 或者协调节点运行
public void start() throws Exception {
SabotContext context = this.context.get();
this.allocator = bufferAllocator.get().newChildAllocator("catalog-protocol", 0, Long.MAX_VALUE);
this.systemNamespace = context.getNamespaceService(SystemUser.SYSTEM_USERNAME);
this.sourceDataStore = kvStoreProvider.get().getStore(CatalogSourceDataCreator.class);
this.modifiableSchedulerService = new ModifiableLocalSchedulerService(1, "modifiable-scheduler-",
ExecConstants.MAX_CONCURRENT_METADATA_REFRESHES, optionManager.get());
this.plugins = newPluginsManager();
plugins.start();
this.protocol = new CatalogProtocol(allocator, new CatalogChangeListener(), config.getSabotConfig());
tunnelFactory = fabric.get().registerProtocol(protocol);
boolean isDistributedCoordinator = config.isMasterlessEnabled()
&& roles.contains(Role.COORDINATOR);
if(roles.contains(Role.MASTER) || isDistributedCoordinator) {
final CountDownLatch wasRun = new CountDownLatch(1);
final Cancellable task = scheduler.get().schedule(ScheduleUtils.scheduleToRunOnceNow(
LOCAL_TASK_LEADER_NAME), () -> {
try {
if (createSourceIfMissing(new SourceConfig()
.setConfig(new InfoSchemaConf().toBytesString())
.setName("INFORMATION_SCHEMA")
.setType("INFORMATION_SCHEMA")
.setMetadataPolicy(CatalogService.NEVER_REFRESH_POLICY))) {
logger.debug("Refreshing 'INFORMATION_SCHEMA' source");
try {
refreshSource(new NamespaceKey("INFORMATION_SCHEMA"), CatalogService.NEVER_REFRESH_POLICY, UpdateType.FULL);
} catch (NamespaceException e) {
throw new RuntimeException(e);
}
}
if (optionManager.get().getOption(ExecConstants.ENABLE_SYSFLIGHT_SOURCE)) {
if (sysFlightTableConfProvider != null) {
logger.info("Creating SysFlight source plugin.");
// check if the sys plugin config type is matching as expected with the flight config type {SYSFLIGHT},
// if not, delete and recreate with flight config.
if (getPlugins().get(SYSTEM_TABLE_SOURCE_NAME) != null
&& !getPlugins().get(SYSTEM_TABLE_SOURCE_NAME).getConnectionConf().getType().equals(sysFlightTableConfProvider.get().get().getType())) {
deleteSource(SYSTEM_TABLE_SOURCE_NAME);
}
createSourceIfMissing(new SourceConfig()
.setConnectionConf(sysFlightTableConfProvider.get().get())
.setName(SYSTEM_TABLE_SOURCE_NAME)
.setMetadataPolicy(CatalogService.DEFAULT_METADATA_POLICY));
}
} else {
// check if the sys plugin config type is matching as expected with the old system table config type {SYS},
// if not, delete and recreate with old system table config type.
if (getPlugins().get(SYSTEM_TABLE_SOURCE_NAME) != null
&& !getPlugins().get(SYSTEM_TABLE_SOURCE_NAME).getConnectionConf().getType().equals(sysTableConfProvider.get().get().getType())) {
deleteSource(SYSTEM_TABLE_SOURCE_NAME);
}
createSourceIfMissing(new SourceConfig()
.setConnectionConf(sysTableConfProvider.get().get())
.setName(SYSTEM_TABLE_SOURCE_NAME)
.setMetadataPolicy(CatalogService.NEVER_REFRESH_POLICY));
}
logger.debug("Refreshing {} source", SYSTEM_TABLE_SOURCE_NAME);
try {
refreshSource(new NamespaceKey(SYSTEM_TABLE_SOURCE_NAME), CatalogService.NEVER_REFRESH_POLICY, UpdateType.FULL);
} catch (NamespaceException e) {
throw new RuntimeException(e);
}
} finally {
wasRun.countDown();
}
}
);
if (!task.isDone()) {
// wait till task is done only if task leader
wasRun.await();
}
}
}
DACDaemonModule 服务bean 定义
registry.bind(CatalogService.class, new CatalogServiceImpl(
registry.provider(SabotContext.class),
registry.provider(SchedulerService.class),
registry.provider(SystemTablePluginConfigProvider.class),
registry.provider(SysFlightPluginConfigProvider.class),
registry.provider(FabricService.class),
registry.provider(ConnectionReader.class),
registry.provider(BufferAllocator.class),
registry.provider(LegacyKVStoreProvider.class),
registry.provider(DatasetListingService.class),
registry.provider(OptionManager.class),
() -> metadataRefreshInfoBroadcaster,
config,
roles
));
DACDaemon 服务的启动，因为这个服务都实现了dremio 自己的service 接口定义，可以方便的进行服务启动管理
说明
CatalogService 服务还是比较重要的，服务的启动主要在master 以及协调节点上
参考资料
sabot/kernel/src/main/java/com/dremio/exec/catalog/CatalogServiceImpl.java
sabot/kernel/src/main/java/com/dremio/exec/catalog/Catalog.java
dac/backend/src/main/java/com/dremio/dac/daemon/DACDaemon.java
common/src/main/java/com/dremio/service/SingletonRegistry.java
common/src/main/java/com/dremio/service/ServiceRegistry.java
