dremio SplitOrphansCleanerService 简单说明
SplitOrphansCleanerService 目前主要是清理一些孤立的split 服务(主要是对于namespaceservice)
SplitOrphansCleanerService创建
DACDaemonModule 中，可以看出是分布式master 角色启动的（实际就是master 角色或者）
if (isDistributedMaster) {
// Companion service to clean split orphans
registry.bind(SplitOrphansCleanerService.class, new SplitOrphansCleanerService(
registry.provider(SchedulerService.class),
registry.provider(NamespaceService.Factory.class),
registry.provider(OptionManager.class)));
}
服务角色判断
final boolean isMasterless = config.isMasterlessEnabled();
final boolean isCoordinator = config.getBoolean(DremioConfig.ENABLE_COORDINATOR_BOOL);
final boolean isExecutor = config.getBoolean(DremioConfig.ENABLE_EXECUTOR_BOOL);
final boolean isDistributedCoordinator = isMasterless && isCoordinator;
内部处理
使用了dremio 的调度服务，默认是小时级别的处理
- SplitOrphansCleanerService start 参考处理
public void start() throws Exception {
final int splitOrphansCleanPeriodHour = Integer.getInteger(SPLIT_ORPHANS_CLEAN_PERIOD_HOUR_PROPERTY,
DEFAULT_SPLIT_ORPHANS_CLEAN_PERIOD_HOUR);
final int splitOrphansReleaseLeadershipHour = Integer.getInteger(SPLIT_ORPHANS_RELEASE_LEADERSHIP_MS_PROPERTY,
DEFAULT_SPLIT_ORPHANS_RELEASE_LEADERSHIP_HOUR_IN_MS);
final NamespaceService namespaceService = namespaceServiceFactory.get().get(SystemUser.SYSTEM_USERNAME);
boolean metadataAutoExpiration = optionManager.getOption(NamespaceOptions.DATASET_METADATA_USE_SMART_EXPIRY);
PartitionChunkId.SplitOrphansRetentionPolicy policy = metadataAutoExpiration ?
new PartitionChunkId.SplitOrphansRetentionPolicy.SmartExpirationPolicyForSplits(optionManager.getOption(NamespaceOptions.DATASET_METADATA_AUTO_EXPIRE_AFTER_HOURS)) :
PartitionChunkId.SplitOrphansRetentionPolicy.KEEP_VALID_SPLITS;
// 结合元数据过期策略进行定时删除
cleanerTask = scheduler.get().schedule(Schedule.Builder.everyHours(splitOrphansCleanPeriodHour)
.asClusteredSingleton(LOCAL_TASK_LEADER_NAME)
.releaseOwnershipAfter(splitOrphansReleaseLeadershipHour, TimeUnit.MILLISECONDS)
.build(), () -> {
logger.info("Search for expired dataset splits");
final int expired = namespaceService.deleteSplitOrphans(policy,
optionManager.getOption(NamespaceOptions.DATASET_METADATA_CONSISTENCY_VALIDATE));
logger.info("Deleted {} expired/orphan dataset splits", expired);
});
}
- namespaceService deleteSplitOrphans 删除
这部分代码比较多，我简单说明下处理，详细的请阅读源码
遍历所有namespace，核心是对于dataset为PHYSICAL_DATASET,PHYSICAL_DATASET_SOURCE_FILE,PHYSICAL_DATASET_SOURCE_FOLDER 类型的基于元数据策略获取列表，生成一个range 列表
官方的一个说明
range被设置为包含每个数据集的分区块的当前（排他）范围。然后，该函数在分区块存储和验证中存在的所有分区块上进行迭代，分区块属于数据集分区块范围之一。如果没有，则删除该项目。二进制搜索在不插入分区块的范围内提供索引已存在于列表中（这是插入点，请参Collections.binarySort
javadoc），它应该正好位于作为范围项的相应数据集范围之后根据它们的下端点进行排序。在multiSplitStore中过期的任何key 也会在partitionChunkStore中过期。这个事实已经习惯了识别multiSplitStore中的孤立拆分，并同时从两个存储中删除，multiSplitStore的负载较少，因为搜索后只查找此存储中的特定拆分，partitionChunkStore来标识要读取的拆分
删除处理 (NamespaceServiceImpl) 部分处理
// 实际上就是基于已经校验正常的列表，然后删除不在正常列表的PartitionChunkId
private void processSplit(final ArrayList<LegacyKVStore> kvStores, PartitionChunkId id, List<Range<PartitionChunkId>> splitsToRetain, ExpiredSplitsTracker expiredSplitsTracker) {
final int item = Collections.binarySearch(splitsToRetain, Range.singleton(id), PARTITION_CHUNK_RANGE_COMPARATOR);
// we should never find a match since we're searching for a split key and that dataset
// split range endpoints are excluded/not valid split keys
Preconditions.checkState(item < 0);
final int insertionPoint = (-item) - 1;
final int consideredRange = insertionPoint - 1; // since a normal match would come directly after the start range, we need to check the range directly above the insertion point.
boolean deleteSplit = (consideredRange < 0 || !splitsToRetain.get(consideredRange).contains(id));
try {
if (deleteSplit) {
// delete from the kvstore in order
for (int i = 0; i < kvStores.size(); i++) {
deleteSplitWithRetry(kvStores.get(i), id, MAX_DELETE_SPLIT_RETRIES, expiredSplitsTracker);
}
}
} finally {
expiredSplitsTracker.trackPartitionChunk(id, deleteSplit);
}
}
说明
dremio 在namespace 中可能会出现元数据不一致的问题，SplitOrphansCleanerService 核心就是处理namespace 中数据不一致的清理
参考资料
services/namespace/src/main/java/com/dremio/service/namespace/SplitOrphansCleanerService.java
services/namespace/src/main/java/com/dremio/service/namespace/NamespaceServiceImpl.java
services/namespace/src/main/java/com/dremio/service/namespace/PartitionChunkId.java
