dremio 的配置处理简单说明

dremio 的配置基于了typesafe 的config

参考配置

  • 模块级别的配置
    sabot-module.conf
 
dremio.classpath.scanning {
    base.classes += "com.dremio.dac.cmd.upgrade.UpgradeTask"
    packages += "com.dremio.dac"
    packages += "com.dremio.dac.support.SupportService"
    package += "com.dremio.dac.support.QueryLogBundleService"
 
    annotations += "com.dremio.dac.cmd.AdminCommand"
    packages += "com.dremio.dac.cmd"
    packages += "com.dremio.dac.cmd.upgrade"
}
  • 参考配置（基础默认的）
    dremio.conf 运行配置
 
paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"
 
  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}
 
services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true
}
 
 

参考类图

typesafe Config 类图

 

 

公共配置加载处理

DremioConfig 配置

 
public static DremioConfig create(final URL userConfigPath, SabotConfig sabotConfig) {
    Config reference = null;
 
    final ClassLoader[] classLoaders = ClasspathHelper.classLoaders();
    for (ClassLoader classLoader : classLoaders) {
      final URL configUrl = classLoader.getResource(REFERENCE_CONFIG);
      if(configUrl == null){
        continue;
      }
      Preconditions.checkArgument(reference == null, "Attempted to load more than one reference configuration.");
      // 参考配置 dremio-reference.conf 
      reference = ConfigFactory.parseResources(classLoader, REFERENCE_CONFIG);
    }
 
    Preconditions.checkNotNull(reference, "Unable to find the reference configuration.");
 
 
    Config userConfig = null;
 
    if(userConfigPath == null){
 
      for (ClassLoader classLoader : classLoaders) {
       // 默认配置 dremio.conf
        final URL configUrl = classLoader.getResource(DEFAULT_USER_CONFIG);
        if(configUrl == null){
          continue;
        }
        Preconditions.checkArgument(userConfig == null, "Attempted to load more than one user configuration.");
        userConfig = ConfigFactory.parseResources(classLoader, DEFAULT_USER_CONFIG);
      }
 
    } else {
 
      userConfig = ConfigFactory.parseURL(userConfigPath, ConfigParseOptions.defaults().setAllowMissing(false));
    }
 
    final Config effective;
 
    if(userConfig != null){
 
      effective = userConfig;
    } else {
      effective = reference;
    }
 
    final Config skinned =
        applySystemProperties(
          applyLegacySystemProperties(effective),
          reference);
 
    return new DremioConfig(sabotConfig, skinned, reference, determineNode());
  }

SabotConfig 配置加载

sabot-override.conf 加载处理

private static SabotConfig doCreate(String overrideFileResourcePathname,
                                      Properties overriderProps) {
    final StringBuilder logString = new StringBuilder();
    final Stopwatch watch = Stopwatch.createStarted();
    overrideFileResourcePathname =
        overrideFileResourcePathname == null
            ? CommonConstants.CONFIG_OVERRIDE_RESOURCE_PATHNAME
            : overrideFileResourcePathname;
 
    // 1. Load defaults configuration file.
    Config fallback = null;
    final ClassLoader[] classLoaders = ClasspathHelper.classLoaders();
    for (ClassLoader classLoader : classLoaders) {
     //  加载默认的
      final URL url =
          classLoader.getResource(CommonConstants.CONFIG_DEFAULT_RESOURCE_PATHNAME);
      if (null != url) {
        logString.append("Base Configuration:\n\t- ").append(url).append("\n");
        fallback =
            ConfigFactory.load(classLoader,
                               CommonConstants.CONFIG_DEFAULT_RESOURCE_PATHNAME);
        break;
      }
    }
    // 加载模块的配置
    // 2. Load per-module configuration files.
    final Collection<URL> urls = ClassPathScanner.getConfigURLs();
    logString.append("\nIntermediate Configuration and Plugin files, in order of precedence:\n");
    for (URL url : urls) {
      logString.append("\t- ").append(url).append("\n");
      fallback = ConfigFactory.parseURL(url).withFallback(fallback);
    }
    logString.append("\n");
    // 基于环境变量的配置
    // 3. Load any specified overrides configuration file along with any
    //    overrides from JVM system properties (e.g., {-Dname=value").
 
    // (Per ConfigFactory.load(...)'s mention of using Thread.getContextClassLoader():)
    final URL overrideFileUrl =
        Thread.currentThread().getContextClassLoader().getResource(overrideFileResourcePathname);
    if (null != overrideFileUrl ) {
      logString.append("Override File: ").append(overrideFileUrl).append("\n");
    }
    Config effectiveConfig =
        ConfigFactory.load(overrideFileResourcePathname).withFallback(fallback);
 
    // 4. Apply any overriding properties.
    if (overriderProps != null) {
      logString.append("Overridden Properties:\n");
      for(Entry<Object, Object> entry : overriderProps.entrySet()){
        logString.append("\t-").append(entry.getKey()).append(" = ").append(entry.getValue()).append("\n");
      }
      logString.append("\n");
      effectiveConfig =
          ConfigFactory.parseProperties(overriderProps).withFallback(effectiveConfig);
    }
 
    // 5. Create SabotConfig object from Config object.
    logger.info("Configuration and plugin file(s) identified in {}ms.\n{}",
        watch.elapsed(TimeUnit.MILLISECONDS),
        logString);
    return new SabotConfig(effectiveConfig.resolve());
  }

说明

dremio 基于了typesafe 的config提供了强大的配置管理是一个值得借鉴的设计

参考资料

common/src/main/java/com/dremio/common/config/SabotConfig.java
common/src/main/java/com/dremio/config/DremioConfig.java
common/src/main/java/com/dremio/common/scanner/ClassPathScanner.java
common/src/main/java/com/dremio/common/config/CommonConstants.java
https://github.com/lightbend/config

posted on 2023-01-19 15:54  荣锋亮  阅读(75)  评论(0)  编辑  收藏  举报

导航