dremio yarn 集成之 AppBundleGenerator 简单说明

AppBundleGenerator 的目的是方便yarn 应用的运行，dremio 自己开发了一个方便软件打包的服务，可以简化

参考处理

默认生成的jar 包名称dremio-bundle.jar

public Path generateBundle() throws IOException {

    // Create an application bundle jar based on current application classpath

    Path yarnBundledJarPath = Files.createTempFile(DREMIO_BUNDLE_PREFIX, ".jar");

    // dremio包装的JarGenerator ，生成jar 包

    try (JarGenerator jarGenerator = JarGenerator.of(new JarOutputStream(Files.newOutputStream(yarnBundledJarPath)))) {

      // First add prefix classpath entries

      // Second, add content of classpath to bundle jar

      // Then add extra classpath entries

      List<URI> jarEntries;

      try (Stream<Path> prefixStream = toPathStream(classPathPrefix);

        Stream<Path> classLoaderStream = toPathStream(classLoader);

        Stream<Path> classPathStream = toPathStream(classPath)) {

        jarEntries = addPathsToBundle(jarGenerator,

          Stream.concat(prefixStream, Stream.concat(classLoaderStream, classPathStream)));

      }

 

      // After that add native libraries

      List<URI> nativeLibrariesEntries = addPathsToBundle(jarGenerator, nativeLibraryPath.stream().map(Paths::get));

 

      // After that add plugins

      URI pluginsPathEntry = addPathToJar(jarGenerator, pluginsPath);

     // jar Manifest 添加

      // Finally, add classpath and native library path entries in jar manifest

      // Following spec for class-path, string is a list of URI separated by space...

      Manifest manifest = new Manifest();

      final Attributes mainAttributes = manifest.getMainAttributes();

      mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0");

      mainAttributes.put(Attributes.Name.CLASS_PATH,

          jarEntries.stream()

          .map(URI::toString)

          .collect(Collectors.joining(DELIMITER)));

      mainAttributes.putValue(X_DREMIO_LIBRARY_PATH_MANIFEST_ATTRIBUTE,

          nativeLibrariesEntries.stream().map(URI::toString).collect(Collectors.joining(DELIMITER)));

      mainAttributes.putValue(X_DREMIO_PLUGINS_PATH_MANIFEST_ATTRIBUTE, pluginsPathEntry.toString());

 

      jarGenerator.addManifest(manifest);

    }

 

    return yarnBundledJarPath;

  }

使用

DacDaemonYarnApplication 类的构造函数中

public DacDaemonYarnApplication(DremioConfig dremioConfig, YarnConfiguration yarnConfig, @NotNull Environment env) {

    this.yarnConfig = yarnConfig;

   // 注意需要一个DREMIO_HOME 环境变量

    final String dremioHome = Preconditions.checkNotNull(env.getEnv(DREMIO_HOME),

        "Environment variable DREMIO_HOME is not set");

 

    this.keytabFileLocation = dremioConfig.getString(DremioConfig.KERBEROS_KEYTAB_PATH);

 

    // Gather the list of jars to be added to the container classpath

    Stream<String> classpathJars = Stream.concat(

        yarnConfig.getTrimmedStringCollection(YarnDefaultsConfigurator.CLASSPATH_JARS).stream().map(d -> dremioHome.concat(d)),

        dremioConfig.getStringList(DremioConfig.YARN_CLASSPATH).stream());

 

 

    classpathJars.forEach(classpathJar -> {

      Path jarFullPath = Paths.get(classpathJar);

      Path parentDir = jarFullPath.getParent();

      final String fileName = jarFullPath.getFileName().toString();

 

      try(Stream<Path> paths = Files.list(parentDir)) {

        paths

        .filter(p -> p.getFileName().toString().matches(fileName))

        .forEach(p -> {

          jarNames.add(p.getFileName().toString());

          this.classpathJarNames.add(p.toFile());

        });

      } catch(IOException e) {

        logger.warn("Cannot list files in directory {}", parentDir, e);

      }

    });

 

    // Create an application bundle jar based on current application classpath

    AppBundleGenerator appBundleGenerator = AppBundleGenerator.of(dremioConfig);

    try {

      if (!isTestingModeOn) {

        yarnBundledJarPath = appBundleGenerator.generateBundle();

      } else {

        yarnBundledJarPath = Paths.get("/temp");

      }

    } catch (IOException e) {

      throw new RuntimeException(e);

    }

  }

说明

dremio 的AppBundleGenerator 主要是生成jar 文件（都使用了java jar 操作api），方便yarn 运行（实际上就是fat jar），后边YarnController 会使用到，具体会介绍

参考资料

provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/AppBundleGenerator.java
provision/yarn/yarntwill/src/main/java/com/dremio/provision/yarn/DacDaemonYarnApplication.java
https://docs.dremio.com/software/advanced-administration/start-stop/#starting-up-with-yarn-deployments
https://docs.dremio.com/software/deployment/yarn-hadoop/