dremio ClassPathScanner 简单说明

ClassPathScanner 实际上是dremio 基于配置+约定的类扫描处理，为了性能同时使用了开源的reflections 工具包

约定

主要扫描包含以下约定key 的类或者包

 /** Configuration pathname to list of names of packages to scan for implementations. */
  private static final String IMPLEMENTATIONS_SCAN_PACKAGES = "dremio.classpath.scanning.packages";
 
  /** Configuration pathname to list of names of base classes to scan for implementations. */
  private static final String IMPLEMENTATIONS_SCAN_CLASSES = "dremio.classpath.scanning.base.classes";
 
  /** Configuration pathname to list of names of annotations to scan for. */
  private static final String IMPLEMENTATIONS_SCAN_ANNOTATIONS = "dremio.classpath.scanning.annotations";
 
  /** Configuration pathname to turn off build time caching. */
  private static final String IMPLEMENTATIONS_SCAN_CACHE = "dremio.classpath.scanning.cache.enabled";

配置

基于了typesafe.config ，参考配置文件sabot-module.conf

dremio {
  classpath.scanning {
 
    base.classes : ${?dremio.classpath.scanning.base.classes} [
      com.dremio.common.logical.data.LogicalOperator,
      com.dremio.common.logical.FormatPluginConfig,
      com.dremio.common.store.StoragePluginConfig,
      com.dremio.service.Initializer
    ],
 
    packages : ${?dremio.classpath.scanning.packages} [
      com.dremio.exec.store.mock,
      com.dremio.common.logical,
      com.dremio.exec.store.dfs,
      com.dremio.exec.server.options,
      com.dremio.plugins.mongo
    ]
  },
 
  memory: {
      debug.error_on_leak: true,
      top.max: 1000000000000
  }
}

参考使用

会返回一个ScanResult

public static ScanResult fromPrescan(SabotConfig config) {
    List<String> packagePrefixes = ClassPathScanner.getPackagePrefixes(config);
    List<String> scannedBaseClasses = ClassPathScanner.getScannedBaseClasses(config);
    List<String> scannedAnnotations = ClassPathScanner.getScannedAnnotations(config);
    if (ClassPathScanner.isScanBuildTimeCacheEnabled(config)) {
      // scan only locations that have not been scanned yet
      ScanResult runtimeScan = ClassPathScanner.scan(
          NON_PRESCANNED_MARKED_PATHS,
          packagePrefixes,
          scannedBaseClasses,
          scannedAnnotations,
          PRESCANNED);
      return runtimeScan.merge(PRESCANNED);
    } else {
      // scan everything
      return ClassPathScanner.scan(
          ClassPathScanner.getMarkedPaths(),
          packagePrefixes,
          scannedBaseClasses,
          scannedAnnotations,
          ClassPathScanner.emptyResult());
    }
  }

说明

dremio 对于插件的加载就使用了动态创建以及类扫描的处理，灵活性还是很不错的

参考资料

common/src/main/java/com/dremio/common/scanner/ClassPathScanner.java
common/src/main/java/com/dremio/common/config/SabotConfig.java
https://github.com/ronmamo/reflections

posted on 2023-01-02 21:37  荣锋亮  阅读(28)  评论(0)  编辑  收藏  举报

导航