dremio 自动提升分片字段处理简单说明
dremio自动提升支持自动将文件夹转换为一个列，同时可以实现数据的过滤查询，是一个很不错的功能，比如我们在一些数据归档类的应用中可以基于事件进行分区，之后通过自动提升可以方便的进行数据查询
效果
- 查询效果
- debug 效果
内部处理
对于dremio 自动提升的处理我已经介绍过了，实际上核心是分区字段的处理（对应的查询上就是表元数据）
- 保存处理
DatasetManager 中的getTableFromPlugin 方法
try {
plugin.getSaver()
.save(datasetConfig, handle.get(), plugin.unwrap(StoragePlugin.class), opportunisticSave, retrievalOptions,
userName);
} catch (ConcurrentModificationException cme) {
// Some other query, or perhaps the metadata refresh, must have already created this dataset. Re-obtain it
// from the namespace
assert opportunisticSave : "Non-opportunistic saves should have already handled a CME";
try {
datasetConfig = userNamespaceService.getDataset(canonicalKey);
} catch (NamespaceException e) {
// We got a concurrent modification exception because a dataset existed. It shouldn't be the case that it
// no longer exists. In the very rare case of this code racing with both another update *and* a dataset deletion
// we should act as if the delete won
logger.warn("Unable to obtain dataset {}. Likely race with dataset deletion", canonicalKey);
return null;
}
final NamespaceTable namespaceTable = getTableFromNamespace(key, datasetConfig, plugin, accessUserName, options);
options.getStatsCollector()
.addDatasetStat(canonicalKey.getSchemaPath(), MetadataAccessType.CACHED_METADATA.name(),
stopwatch.elapsed(TimeUnit.MILLISECONDS));
return namespaceTable;
} catch (AccessControlException ignored) {
return null;
}
options.getStatsCollector()
.addDatasetStat(canonicalKey.getSchemaPath(), MetadataAccessType.PARTIAL_METADATA.name(),
stopwatch.elapsed(TimeUnit.MILLISECONDS));
plugin.checkAccess(canonicalKey, datasetConfig, accessUserName, options);
// TODO: use MaterializedSplitsPointer if metadata is not too big!
final TableMetadata tableMetadata = new TableMetadataImpl(plugin.getId(), datasetConfig,
accessUserName, DatasetSplitsPointer.of(userNamespaceService, datasetConfig),
getPrimaryKey(plugin.getPlugin(), datasetConfig, schemaConfig, key,
false /* The metadata for the table is either incomplete, missing or out of date.
Storing in namespace can cause issues if the metadata is missing. Don't save here.
*/));
return new NamespaceTable(tableMetadata, optionManager.getOption(FULL_NESTED_SCHEMA_SUPPORT));
}
- 内部保存对于datasetconf 的扩展
DatasetSaverImpl saveUsingV1Flow
private void saveUsingV1Flow(DatasetHandle handle,
DatasetRetrievalOptions options,
DatasetConfig datasetConfig,
SourceMetadata sourceMetadata,
boolean opportunisticSave,
Function<DatasetConfig, DatasetConfig> datasetMutator,
NamespaceKey canonicalKey,
NamespaceAttribute
if (datasetConfig.getId() == null) {
// this is a new dataset, otherwise save will fail
datasetConfig.setId(new EntityId(UUID.randomUUID().toString()));
}
NamespaceService.SplitCompression splitCompression = NamespaceService.SplitCompression.valueOf(optionManager.getOption(CatalogOptions.SPLIT_COMPRESSION_TYPE).toUpperCase());
try (DatasetMetadataSaver saver = systemNamespace.newDatasetMetadataSaver(canonicalKey, datasetConfig.getId(), splitCompression,
optionManager.getOption(CatalogOptions.SINGLE_SPLIT_PARTITION_MAX), optionManager.getOption(NamespaceOptions.DATASET_METADATA_CONSISTENCY_VALIDATE))) {
final PartitionChunkListing chunkListing = sourceMetadata.listPartitionChunks(handle,
options.asListPartitionChunkOptions(datasetConfig));
final long recordCountFromSplits = saver == null || chunkListing == null ? 0 :
CatalogUtil.savePartitionChunksInSplitsStores(saver,chunkListing);
// 此处会基于实际的dataset handle 进行数据集元数据的获取，比如txt，json 格式的EasyFormatDatasetAccessor handle
final DatasetMetadata datasetMetadata = sourceMetadata.getDatasetMetadata(handle, chunkListing,
options.asGetMetadataOptions(datasetConfig));
Optional<ByteString> readSignature = Optional.empty();
if (sourceMetadata instanceof SupportsReadSignature) {
final BytesOutput output = ((SupportsReadSignature) sourceMetadata).provideSignature(handle, datasetMetadata);
//noinspection ObjectEquality
if (output != BytesOutput.NONE) {
readSignature = Optional.of(MetadataObjectsUtils.toProtostuff(output));
}
}
// 处理会扩展分区信息
MetadataObjectsUtils.overrideExtended(datasetConfig, datasetMetadata, readSignature,
recordCountFromSplits, options.maxMetadataLeafColumns());
datasetConfig = datasetMutator.apply(datasetConfig);
saver.saveDataset(datasetConfig, opportunisticSave, attributes);
updateListener.metadataUpdated(canonicalKey);
} catch (DatasetMetadataTooLargeException e) {
datasetConfig.setRecordSchema(null);
datasetConfig.setReadDefinition(null);
try {
systemNamespace.addOrUpdateDataset(canonicalKey, datasetConfig);
} catch (NamespaceException ignored) {
}
throw UserException.validationError(e)
.build(logger);
} catch (NamespaceException| IOException e) {
throw UserException.validationError(e)
.build(logger);
}
}
}
- EasyFormatDatasetAccessor 参考实现
getBatchSchema 是一个核心实现，进行了扩展
private BatchSchema getBatchSchema(BatchSchema oldSchema, final FileSelection selection, final FileSystem dfs) throws Exception {
final SabotContext context = formatPlugin.getContext();
try (
BufferAllocator sampleAllocator = context.getAllocator().newChildAllocator("sample-alloc", 0, Long.MAX_VALUE);
OperatorContextImpl operatorContext = new OperatorContextImpl(context.getConfig(), context.getDremioConfig(), sampleAllocator, context.getOptionManager(), 1000, context.getExpressionSplitCache());
SampleMutator mutator = new SampleMutator(sampleAllocator)
) {
// 查找隐藏的列信息
final ImplicitFilesystemColumnFinder explorer = new ImplicitFilesystemColumnFinder(context.getOptionManager(),
dfs, GroupScan.ALL_COLUMNS, ImplicitFilesystemColumnFinder.Mode.ALL_IMPLICIT_COLUMNS);
Optional<FileAttributes> fileName = selection.getFileAttributesList().stream().filter(input -> input.size() > 0).findFirst();
final FileAttributes file = fileName.orElse(selection.getFileAttributesList().get(0));
EasyDatasetSplitXAttr dataset = EasyDatasetSplitXAttr.newBuilder()
.setStart(0L)
.setLength(Long.MAX_VALUE)
.setPath(file.getPath().toString())
.build();
try (RecordReader reader = new AdditionalColumnsRecordReader(operatorContext, ((EasyFormatPlugin) formatPlugin)
.getRecordReader(operatorContext, dfs, dataset, GroupScan.ALL_COLUMNS), explorer.getImplicitFieldsForSample(selection), sampleAllocator)) {
reader.setup(mutator);
Map<String, ValueVector> fieldVectorMap = new HashMap<>();
int i = 0;
for (VectorWrapper<?> vw : mutator.getContainer()) {
fieldVectorMap.put(vw.getField().getName(), vw.getValueVector());
if (++i > maxLeafColumns) {
throw new ColumnCountTooLargeException(maxLeafColumns);
}
}
reader.allocate(fieldVectorMap);
reader.next();
mutator.getContainer().buildSchema(BatchSchema.SelectionVectorMode.NONE);
// 进行采样的显式字段以及隐式的schema 合并，其中就有我们说的文件夹隐式字段
return getMergedSchema(oldSchema,
mutator.getContainer().getSchema(),
oldConfig.isSchemaLearningEnabled(),
oldConfig.getDropColumns(),
oldConfig.getModifiedColumns(),
context.getOptionManager().getOption(ExecConstants.ENABLE_INTERNAL_SCHEMA),
tableSchemaPath.getPathComponents(),
file.getPath().toString());
}
}
}
一个小技巧
从我上边的图中可以看出还有几个隐藏字段也可以实现（文件名以及修改时间），是可以通过配置显示的
- 相关配置
ImplicitFilesystemColumnFinder
public static final StringValidator IMPLICIT_FILE_FIELD_LABEL = new StringValidator("dremio.store.file.file-field-label", "$file");
public static final StringValidator IMPLICIT_MOD_FIELD_LABEL = new StringValidator("dremio.store.file.mod-field-label", "$mtime");
public static final BooleanValidator IMPLICIT_FILE_FIELD_ENABLE = new BooleanValidator("dremio.store.file.file-field-enabled", false);
public static final BooleanValidator IMPLICIT_DIRS_FIELD_ENABLE = new BooleanValidator("dremio.store.file.dir-field-enabled", true);
public static final BooleanValidator IMPLICIT_MOD_FIELD_ENABLE = new BooleanValidator("dremio.store.file.mod-field-enabled", false);
说明
以上是一个关于自动提升分区字段的简单说明，详细的可以参考源码学习，我只是简单介绍了EasyFormatDatasetAccessor，当然还会有其他一些实现
参考资料
sabot/kernel/src/main/java/com/dremio/exec/catalog/DatasetManager.java
sabot/kernel/src/main/java/com/dremio/exec/store/dfs/implicit/AdditionalColumnsRecordReader.java
sabot/kernel/src/main/java/com/dremio/exec/store/easy/EasyFormatDatasetAccessor.java
sabot/kernel/src/main/java/com/dremio/exec/catalog/DatasetSaverImpl.java
sabot/kernel/src/main/java/com/dremio/exec/store/dfs/implicit/ImplicitFilesystemColumnFinder.java
sabot/kernel/src/main/java/com/dremio/exec/util/MetadataSupportsInternalSchema.java
sabot/kernel/src/main/java/com/dremio/exec/store/dfs/implicit/ImplicitFilesystemColumnFinder.java
