dremio web ui手工format 处理简单说明
以前我介绍过通过自动提升处理format,当然dremio也支持手工操作,对于处理包含了文件以及文件夹类型的,我主要说下
关于文件夹的
处理简单说明
对于ui部分是首先进行预处理,获取可能的类型格式,然后就是通过格式预览服务进行数据处理
folder_format参考处理
- 代码
包含了一个快速以及默认,默认配置是快速检测
@GET
@Path("/folder_format/{path: .*}")
@Produces(MediaType.APPLICATION_JSON)
public FileFormatUI getFolderFormat(@PathParam("path") String path)
throws PhysicalDatasetNotFoundException, NamespaceException, SourceNotFoundException, IOException {
if (useFastPreview()) {
SourceFolderPath folderPath = asFolderPath(path);
// 使用formatTools 工具类处理
return new FileFormatUI(formatTools.getOrDetectFormat(folderPath, DatasetType.PHYSICAL_DATASET_SOURCE_FOLDER), folderPath);
}
SourceFolderPath folderPath = SourceFolderPath.fromURLPath(sourceName, path);
sourceService.checkSourceExists(folderPath.getSourceName());
FileFormat fileFormat;
try {
final PhysicalDatasetConfig physicalDatasetConfig = sourceService.getFilesystemPhysicalDataset(folderPath);
fileFormat = FileFormat.getForFolder(physicalDatasetConfig.getFormatSettings());
fileFormat.setVersion(physicalDatasetConfig.getTag());
} catch (PhysicalDatasetNotFoundException nfe) {
fileFormat = sourceService.getDefaultFileFormat(sourceName, folderPath, securityContext.getUserPrincipal().getName());
}
return new FileFormatUI(fileFormat, folderPath);
}
- FormatTools getOrDetectFormat处理
public FileFormat getOrDetectFormat(NamespacePath folderPath, DatasetType expectedType) throws NamespaceException {
// folder
final FileFormat fileFormat;
try {
final PhysicalDatasetConfig physicalDatasetConfig = sourceService.getFilesystemPhysicalDataset(folderPath, expectedType);
if (!expectedType.equals(physicalDatasetConfig.getType())) {
throw new IllegalStateException(
String.format(
"Expected format of type %s but actually of format %s.",
expectedType,
physicalDatasetConfig.getType()
)
);
}
// determine whether folder or file.
final boolean isFolder;
final DatasetType datasetType = physicalDatasetConfig.getType();
Span.current().setAttribute("formattools.getOrDetectFormat.datasetType", datasetType.name());
switch(datasetType) {
case PHYSICAL_DATASET_HOME_FILE:
case PHYSICAL_DATASET_SOURCE_FILE:
isFolder = false;
break;
case PHYSICAL_DATASET_SOURCE_FOLDER:
case PHYSICAL_DATASET_HOME_FOLDER:
isFolder = true;
break;
case PHYSICAL_DATASET:
case VIRTUAL_DATASET:
default:
throw new IllegalStateException("Dataset is neither a file nor a folder.");
}
final FileConfig fileConfig = physicalDatasetConfig.getFormatSettings();
// 核心是基于FileFormat 工具类处理的
fileFormat = isFolder ? FileFormat.getForFolder(fileConfig) : FileFormat.getForFile(fileConfig);
fileFormat.setVersion(physicalDatasetConfig.getTag());
return fileFormat;
} catch (PhysicalDatasetNotFoundException nfe) {
// ignore and fall through to detect the format, so we don't have extra nested blocks.
}
final NamespaceKey key = folderPath.toNamespaceKey();
return detectFileFormat(key);
}
- FileFormat get 方法
private static FileFormat get(FileConfig fileConfig) {
// TODO (Amit H) Remove after defining classes for tsv, csv, and psv
FileType fileType = fileConfig.getType();
if (fileType == FileType.CSV || fileType == FileType.TSV || fileType == FileType.PSV) {
fileType = FileType.TEXT;
}
final Class<? extends FileFormat> fileFormatClass = FileFormatDefinitions.CLASS_TYPES.get(fileType);
final Schema<FileFormat> schema = (Schema<FileFormat>) FileFormatDefinitions.SCHEMAS.get(fileFormatClass);
final FileFormat fileFormat = schema.newMessage();
if (fileConfig.getExtendedConfig() != null) {
ProtobufIOUtil.mergeFrom(fileConfig.getExtendedConfig().toByteArray(), fileFormat, schema);
}
fileFormat.setCtime(fileConfig.getCtime());
fileFormat.setName(fileConfig.getName());
fileFormat.setOwner(fileConfig.getOwner());
fileFormat.setFullPath(fileConfig.getFullPathList());
fileFormat.setVersion(fileConfig.getTag());
fileFormat.setLocation(fileConfig.getLocation());
return fileFormat;
}
- 接口返回格式
参考如下
{
"fileFormat": {
"type": "Iceberg", // 类型
"name": "NYC-taxi-trips-iceberg",
"fullPath": [
"Samples",
"samples.dremio.com",
"NYC-taxi-trips-iceberg"
],
"ctime": 1708002674824,
"isFolder": true
},
"links": {
"format_preview": "/source/Samples/folder_preview/samples.dremio.com/NYC-taxi-trips-iceberg", // 格式预览地址
"self": "/source/Samples/folder_format/samples.dremio.com/NYC-taxi-trips-iceberg"
},
"id": "/source/Samples/folder_format/samples.dremio.com/NYC-taxi-trips-iceberg"
}
- 格式预览处理
protected JobDataFragment previewFormat(FileFormat format, NamespacePath path) {
if (useFastPreview()) {
try {
// 与格式预处理类似也提供了快速以及默认,默认是快速
return formatTools.previewData(format, path, false);
} catch (AccessControlException e) {
throw UserException.validationError()
.message(e.getMessage())
.buildSilently();
}
}
SourceFilePath filePath = SourceFilePath.fromURLPath(sourceName, path.toPathString());
return executor.previewPhysicalDataset(filePath.toString(), format, getOrCreateAllocator("previewFileFormat"));
}
- 快速预览使用了格式工具类处理的
实际上是通过格式插件处理的,dremio 实现了不同数据类型的处理,直接读取的是文件系统的数据
public JobDataFragment previewData(FileFormat format, NamespacePath namespacePath, boolean useFormatLocation) {
final NamespaceKey key = namespacePath.toNamespaceKey();
final FileSystemPlugin<?> plugin = getPlugin(key);
FileSystem fs;
try {
fs = plugin.createFS(securityContext.getUserPrincipal().getName());
} catch (IOException ex) {
throw new IllegalStateException("No files detected or unable to read data.", ex);
}
final Path path = FileSelection.getPathBasedOnFullPath(plugin.resolveTableNameToValidPath(key.getPathComponents()));
// for now, existing rudimentary behavior that uses extension detection.
final FileAttributes attributes;
try {
attributes = fs.getFileAttributes(path);
} catch(IOException ex) {
// we could return unknown but if there are no files, what's the point.
throw new IllegalStateException("No files detected or unable to read data.", ex);
}
try {
final FormatPluginConfig formatPluginConfig = PhysicalDatasetUtils.toFormatPlugin(format.asFileConfig(), Collections.<String>emptyList());
final FormatPlugin formatPlugin = plugin.getFormatPlugin(formatPluginConfig);
if(attributes.isRegularFile()) {
return getData(formatPlugin, fs, Collections.singleton(attributes).iterator());
}
try(DirectoryStream<FileAttributes> files = formatPlugin.getFilesForSamples(fs, plugin, path)) {
Iterator<FileAttributes> iter = files.iterator();
return getData(formatPlugin, fs, Iterators.filter(iter, FileAttributes::isRegularFile));
}
} catch (DirectoryIteratorException ex) {
throw new RuntimeException(ex.getCause());
} catch (Exception ex) {
throw Throwables.propagate(ex);
}
}
- 默认预览处理
默认的处理是通过QueryExecutor 的previewPhysicalDataset 处理的,内部是通过JobsService 接口的实现处理的
public JobDataFragment previewPhysicalDataset(String table, FileFormat formatOptions, BufferAllocator allocator) {
// 注意此处因为没有元数据信息,所以dremio 对于sql 查询使用了一个table 表函数(macro)进行处理,这样就不需要metadata 了,不然dremio 会提示表找不到
final com.dremio.service.job.SqlQuery query = JobRequestUtil.createSqlQuery(format("select * from table(%s (%s))", table, formatOptions.toTableOptions()),
null, context.getUserPrincipal().getName());
// We still need to truncate the results to 500 as the preview physical datasets doesn't support pagination yet
final CompletionListener listener = new CompletionListener();
final JobSubmission jobSubmission = jobsService.submitJob(
SubmitJobRequest.newBuilder().setSqlQuery(query).setQueryType(com.dremio.service.job.QueryType.UI_INITIAL_PREVIEW).build(),
listener);
listener.awaitUnchecked();
return new JobDataWrapper(jobsService, jobSubmission.getJobId(), jobSubmission.getSessionId(), SystemUser.SYSTEM_USERNAME)
.truncate(allocator, 500);
}
}
- 表函数生成的sql参考
一个示例(这样就可以直接查询了,就不直接需要metadata 了)
select * from table(s3v2.data (type => 'text', fieldDelimiter => ',', comment => '#', "escape" => '"', quote => '"', lineDelimiter => '
', extractHeader => false, skipFirstLine => false, autoGenerateColumnNames => true, trimHeader => true))
- JobsService 处理
实际上是发起了一个grpc 请求,获取数据
public JobSubmission submitJob(SubmitJobRequest jobRequest, JobStatusListener statusListener) {
final JobStatusListenerAdapter adapter = new JobStatusListenerAdapter(statusListener);
getAsyncStub().submitJob(jobRequest, adapter);
JobSubmission jobSubmission = adapter.getJobSubmission();
enableTraces(jobSubmission);
return jobSubmission;
}
- 后续处理
预览之后dremio 同时还会发起一个put 请求对于实际dataset 数据格式的更新,实际上操作的就是datasets 的元数据,确保之后查询直接可以作为一个dremio 标准表了
说明
以上是一个简单说明ui 层是如何进行格式化处理的,内部详细的可以阅读源码,format 是dremio ui 层一个很不错的功能,可以实现灵活的数据处理,设计上值得好好学习下
参考资料
dac/backend/src/main/java/com/dremio/dac/resource/SourceResource.java
dac/backend/src/main/java/com/dremio/dac/model/sources/FormatTools.java
dac/backend/src/main/java/com/dremio/dac/service/source/SourceService.java
services/namespace/src/main/java/com/dremio/service/namespace/file/FileFormat.java
services/namespace/src/main/java/com/dremio/service/namespace/file/FileFormatDefinitions.java
sabot/kernel/src/main/java/com/dremio/exec/store/dfs/FormatPlugin.java
dac/backend/src/main/java/com/dremio/dac/explore/QueryExecutor.java
services/jobs/src/main/java/com/dremio/service/jobs/HybridJobsService.java
services/jobs/src/main/java/com/dremio/service/jobs/JobsService.java
services/jobs/target/generated-sources/protobuf/grpc-java/com/dremio/service/job/JobsServiceGrpc.java