dremio SourceCatalog 服务说明
SourceCatalog 主要进行source 的管理，包含了获取信息，创建，更新，删除，包含了不同的实现
SourceCatalog 服务定义
/**
* Interface to perform actions on sources.
*/
// PrivilegeCatalog 可以实现权限处理，具体实际上是SqlGrant 提供的能力
public interface SourceCatalog extends PrivilegeCatalog {
SourceState refreshSourceStatus(NamespaceKey key) throws Exception;
/**
* Get a source based on the provided name. If the source doesn't exist, synchronize with the
* KVStore to confirm creation status.
*
* @param name
* @return A StoragePlugin casted to the expected output.
*/
<T extends StoragePlugin> T getSource(String name);
/**
* Create a source based on the provided configuration. Includes both the creation as well the
* startup of the source. If the source fails to start, no creation will be done. The provided
* configuration should have a null version. Failure to create or failure to start with throw an
* exception. Additionally, if "store.plugin.check_state" is enabled, a plugin that starts but
* then reveals a bad state, will also result in exception.
*
* @param config Configuration for the source.
* @param attributes Optional namespace attributes to pass to namespace entity creation
*/
void createSource(SourceConfig config, NamespaceAttribute
/**
* Update an existing source with the given config. The config version must be the same as the
* currently active source. If it isn't, this call will fail with an exception.
*
* @param config Configuration for the source.
* @param attributes Optional namespace attributes to pass to namespace entity creation
*/
void updateSource(SourceConfig config, NamespaceAttribute
/**
* Delete a source with the provided config. If the source doesn't exist or the config doesn't
* match, the method with throw an exception.
*
* @param config
*/
void deleteSource(SourceConfig config);
}
实现子类
参考下图，还是比较复杂的，实际实现是CatalogImpl，内部会调用CatalogServiceImpl
创建source创建处理
参考处理
private void createSource(SourceConfig config, CatalogIdentity subject, NamespaceAttribute
boolean afterUnknownEx = false;
try(final AutoCloseable sourceDistributedLock = getDistributedLock(config.getName())) {
logger.debug("Obtained distributed lock for source {}", "-source-"+config.getName());
setInfluxSource(config.getName());
// 通过 PluginsManager 插件管理进行实际source 的创建
getPlugins().create(config, subject.getName(), attributes);
communicateChange(config, RpcType.REQ_SOURCE_CONFIG);
} catch (SourceAlreadyExistsException e) {
throw UserException.concurrentModificationError(e).message("Source already exists with name %s.", config.getName()).buildSilently();
} catch (ConcurrentModificationException ex) {
throw ex;
} catch (UserException ue) {
// If it's a UserException, message is probably helpful, so rethrow
throw ue;
} catch (IllegalArgumentException e) {
throw e;
} catch (Exception ex) {
afterUnknownEx = true;
logger.error("Exception encountered: {}", ex.getMessage(), ex);
throw UserException.validationError(ex).message("Failed to create source with name %s.", config.getName()).buildSilently();
} finally {
logger.debug("Releasing distributed lock for source {}", "-source-"+config.getName());
removeInfluxSource(config.getName(), afterUnknownEx);
}
}
PluginsManager start 部分会对于注册插件的获取基于了NamespaceService.Factory
ImmutableMap.Builder<String, CompletableFuture<SourceState>> futuresBuilder = ImmutableMap.builder();
// 此处获取已经注册的插件，转化为dremio 托管的存储插件
for (SourceConfig source : datasetListing.getSources(SystemUser.SYSTEM_USERNAME)) {
ManagedStoragePlugin plugin = newPlugin(source);
futuresBuilder.put(source.getName(), plugin.startAsync());
plugins.put(c(source.getName()), plugin);
}
ConnectionReader 进行插件信息获取
ConnectionReaderImpl 中，方便页面加载
参考处理
protected static Collection<Class<? extends ConnectionConf<?, ?>>> getCandidateSources(ScanResult scanResult) {
ImmutableList.Builder<Class<? extends ConnectionConf<?, ?>>> candidates = new ImmutableList.Builder<>();
for(Class<?> input : scanResult.getAnnotatedClasses(SourceType.class)) {
try {
if (Modifier.isAbstract(input.getModifiers())
|| Modifier.isInterface(input.getModifiers())
|| !ConnectionConf.class.isAssignableFrom(input)) {
logger.warn("Failure trying to recognize SourceConf for {}. Expected a concrete implementation of SourceConf.", input.getName());
continue;
}
} catch (Exception e) {
logger.warn("Failure trying to recognize SourceConf for {}", input.getName(), e);
continue;
}
// Check done just above
candidates.add((Class<? extends ConnectionConf<?, ?>>) input);
}
return candidates.build();
}
说明
source 类型，主要面向的是可以界面可见的存储扩展处理，实际上dremio 还包含了不少内置的存储扩展，系统存储扩展（比如加速，home）
同时通过阅读我们发现官方还是提供了一个PrivilegeCatalog 支持权限能力的扩展的，自己扩展下就可以实现一些企业版的特性了
参考资料
sabot/kernel/src/main/java/com/dremio/exec/catalog/SourceCatalog.java
sabot/kernel/src/main/java/com/dremio/exec/planner/sql/parser/SqlGrant.java
dac/backend/src/main/java/com/dremio/dac/api/SourceResource.java
sabot/kernel/src/main/java/com/dremio/exec/catalog/ConnectionReader.java
sabot/kernel/src/main/java/com/dremio/exec/catalog/ConnectionReaderImpl.java
