dremio yarn 集成之 AppBundleGenerator 简单说明

AppBundleGenerator 的目的是方便yarn 应用的运行，dremio 自己开发了一个方便软件打包的服务，可以简化

参考处理

默认生成的jar 包名称dremio-bundle.jar

public Path generateBundle() throws IOException {
    // Create an application bundle jar based on current application classpath
    Path yarnBundledJarPath = Files.createTempFile(DREMIO_BUNDLE_PREFIX, ".jar");
    // dremio包装的JarGenerator ，生成jar 包
    try (JarGenerator jarGenerator = JarGenerator.of(new JarOutputStream(Files.newOutputStream(yarnBundledJarPath)))) {
      // First add prefix classpath entries
      // Second, add content of classpath to bundle jar
      // Then add extra classpath entries
      List<URI> jarEntries;
      try (Stream<Path> prefixStream = toPathStream(classPathPrefix);
        Stream<Path> classLoaderStream = toPathStream(classLoader);
        Stream<Path> classPathStream = toPathStream(classPath)) {
        jarEntries = addPathsToBundle(jarGenerator,
          Stream.concat(prefixStream, Stream.concat(classLoaderStream, classPathStream)));
      }
 
      // After that add native libraries
      List<URI> nativeLibrariesEntries = addPathsToBundle(jarGenerator, nativeLibraryPath.stream().map(Paths::get));
 
      // After that add plugins
      URI pluginsPathEntry = addPathToJar(jarGenerator, pluginsPath);
     // jar Manifest 添加
      // Finally, add classpath and native library path entries in jar manifest
      // Following spec for class-path, string is a list of URI separated by space...
      Manifest manifest = new Manifest();
      final Attributes mainAttributes = manifest.getMainAttributes();
      mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0");
      mainAttributes.put(Attributes.Name.CLASS_PATH,
          jarEntries.stream()
          .map(URI::toString)
          .collect(Collectors.joining(DELIMITER)));
      mainAttributes.putValue(X_DREMIO_LIBRARY_PATH_MANIFEST_ATTRIBUTE,
          nativeLibrariesEntries.stream().map(URI::toString).collect(Collectors.joining(DELIMITER)));
      mainAttributes.putValue(X_DREMIO_PLUGINS_PATH_MANIFEST_ATTRIBUTE, pluginsPathEntry.toString());
 
      jarGenerator.addManifest(manifest);
    }
 
    return yarnBundledJarPath;
  }

使用

DacDaemonYarnApplication 类的构造函数中

public DacDaemonYarnApplication(DremioConfig dremioConfig, YarnConfiguration yarnConfig, @NotNull Environment env) {
    this.yarnConfig = yarnConfig;
   // 注意需要一个DREMIO_HOME 环境变量
    final String dremioHome = Preconditions.checkNotNull(env.getEnv(DREMIO_HOME),
        "Environment variable DREMIO_HOME is not set");
 
    this.keytabFileLocation = dremioConfig.getString(DremioConfig.KERBEROS_KEYTAB_PATH);
 
    // Gather the list of jars to be added to the container classpath
    Stream<String> classpathJars = Stream.concat(
        yarnConfig.getTrimmedStringCollection(YarnDefaultsConfigurator.CLASSPATH_JARS).stream().map(d -> dremioHome.concat(d)),
        dremioConfig.getStringList(DremioConfig.YARN_CLASSPATH).stream());
 
 
    classpathJars.forEach(classpathJar -> {
      Path jarFullPath = Paths.get(classpathJar);
      Path parentDir = jarFullPath.getParent();
      final String fileName = jarFullPath.getFileName().toString();
 
      try(Stream<Path> paths = Files.list(parentDir)) {
        paths
        .filter(p -> p.getFileName().toString().matches(fileName))
        .forEach(p -> {
          jarNames.add(p.getFileName().toString());
          this.classpathJarNames.add(p.toFile());
        });
      } catch(IOException e) {
        logger.warn("Cannot list files in directory {}", parentDir, e);
      }
    });
 
    // Create an application bundle jar based on current application classpath
    AppBundleGenerator appBundleGenerator = AppBundleGenerator.of(dremioConfig);
    try {
      if (!isTestingModeOn) {
        yarnBundledJarPath = appBundleGenerator.generateBundle();
      } else {
        yarnBundledJarPath = Paths.get("/temp");
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

说明

dremio 的AppBundleGenerator 主要是生成jar 文件（都使用了java jar 操作api），方便yarn 运行（实际上就是fat jar），后边YarnController 会使用到，具体会介绍

参考资料

provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/AppBundleGenerator.java
provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/DacDaemonYarnApplication.java
https://docs.dremio.com/software/advanced-administration/start-stop/#starting-up-with-yarn-deployments
https://docs.dremio.com/software/deployment/yarn-hadoop/

posted on 2023-02-04 21:20  荣锋亮  阅读(18)  评论(0)  编辑  收藏  举报

导航