dremio ClassPathScanner 简单说明

ClassPathScanner 实际上是dremio 基于配置+约定的类扫描处理，为了性能同时使用了开源的reflections 工具包

约定

主要扫描包含以下约定key 的类或者包

 /** Configuration pathname to list of names of packages to scan for implementations. */

  private static final String IMPLEMENTATIONS_SCAN_PACKAGES = "dremio.classpath.scanning.packages";

 

  /** Configuration pathname to list of names of base classes to scan for implementations. */

  private static final String IMPLEMENTATIONS_SCAN_CLASSES = "dremio.classpath.scanning.base.classes";

 

  /** Configuration pathname to list of names of annotations to scan for. */

  private static final String IMPLEMENTATIONS_SCAN_ANNOTATIONS = "dremio.classpath.scanning.annotations";

 

  /** Configuration pathname to turn off build time caching. */

  private static final String IMPLEMENTATIONS_SCAN_CACHE = "dremio.classpath.scanning.cache.enabled";

配置

基于了typesafe.config ，参考配置文件sabot-module.conf

dremio {

  classpath.scanning {

 

    base.classes : ${?dremio.classpath.scanning.base.classes} [

      com.dremio.common.logical.data.LogicalOperator,

      com.dremio.common.logical.FormatPluginConfig,

      com.dremio.common.store.StoragePluginConfig,

      com.dremio.service.Initializer

    ],

 

    packages : ${?dremio.classpath.scanning.packages} [

      com.dremio.exec.store.mock,

      com.dremio.common.logical,

      com.dremio.exec.store.dfs,

      com.dremio.exec.server.options,

      com.dremio.plugins.mongo

    ]

  },

 

  memory: {

      debug.error_on_leak: true,

      top.max: 1000000000000

  }

}

参考使用

会返回一个ScanResult

public static ScanResult fromPrescan(SabotConfig config) {

    List<String> packagePrefixes = ClassPathScanner.getPackagePrefixes(config);

    List<String> scannedBaseClasses = ClassPathScanner.getScannedBaseClasses(config);

    List<String> scannedAnnotations = ClassPathScanner.getScannedAnnotations(config);

    if (ClassPathScanner.isScanBuildTimeCacheEnabled(config)) {

      // scan only locations that have not been scanned yet

      ScanResult runtimeScan = ClassPathScanner.scan(

          NON_PRESCANNED_MARKED_PATHS,

          packagePrefixes,

          scannedBaseClasses,

          scannedAnnotations,

          PRESCANNED);

      return runtimeScan.merge(PRESCANNED);

    } else {

      // scan everything

      return ClassPathScanner.scan(

          ClassPathScanner.getMarkedPaths(),

          packagePrefixes,

          scannedBaseClasses,

          scannedAnnotations,

          ClassPathScanner.emptyResult());

    }

  }

说明

dremio 对于插件的加载就使用了动态创建以及类扫描的处理，灵活性还是很不错的

参考资料

common/src/main/java/com/dremio/common/scanner/ClassPathScanner.java
common/src/main/java/com/dremio/common/config/SabotConfig.java
https://github.com/ronmamo/reflections