dbt click包执行上下文manifest 处理简单说明

dbt 的cli 执行是基于了click 者pip 包，同时为了方便执行（比如依赖的参数），dbt 核心比较重要的是manifest
这个参数是通过context 传递的

run 参考task 处理

• 参考代码
  core dbt/cli/main.py

@cli.command("run")

@click.pass_context

@global_flags

@p.exclude

@p.full_refresh

@p.profile

@p.profiles_dir

@p.project_dir

@p.empty

@p.select

@p.selector

@p.target

@p.target_path

@p.threads

@p.vars

@requires.postflight

@requires.preflight

@requires.profile

@requires.project

@requires.runtime_config

@requires.manifest

def run(ctx, **kwargs):

    """Compile SQL and execute against the current target database."""

    task = RunTask(

        ctx.obj["flags"],

        ctx.obj["runtime_config"],

        ctx.obj["manifest"],

    )

• 简单说明
  cli.command 以及 click.pass_context 就是标准的click 配置p 是配置参数相关的，requires 是必须项，db基于了装饰器进行处理
  @requires.manifest 是一个比较重要的东西dbt 好多命令都是需要此参数的， ctx.obj["manifest"] 此数据是通过装饰器处理的

manifest 参考处理

• core requires.py

def manifest(*args0, write=True, write_perf_info=False):

    """A decorator used by click command functions for generating a manifest

    given a profile, project, and runtime config. This also registers the adapter

    from the runtime config and conditionally writes the manifest to disk.

    """

 

    def outer_wrapper(func):

        def wrapper(*args, **kwargs):

            ctx = args[0]

            assert isinstance(ctx, Context)

 

            req_strs = ["profile", "project", "runtime_config"]

            reqs = [ctx.obj.get(dep) for dep in req_strs]

 

            if None in reqs:

                raise DbtProjectError("profile, project, and runtime_config required for manifest")

 

            runtime_config = ctx.obj["runtime_config"]

 

            # a manifest has already been set on the context, so don't overwrite it

            if ctx.obj.get("manifest") is None:

               # 如果数据为空，会进行解析生成（会结合runtime 配置信息）

                ctx.obj["manifest"] = parse_manifest(

                    runtime_config, write_perf_info, write, ctx.obj["flags"].write_json

                )

            else:

               # 已经包含了需要的信息，会进行adapter 的注册，以及macro 的解析处理

                register_adapter(runtime_config, get_mp_context())

                adapter = get_adapter(runtime_config)

                adapter.set_macro_context_generator(generate_runtime_macro_context)

                adapter.set_macro_resolver(ctx.obj["manifest"])

                query_header_context = generate_query_header_context(

                    adapter.config, ctx.obj["manifest"]

                )

                adapter.connections.set_query_header(query_header_context)

            return func(*args, **kwargs)

 

        return update_wrapper(wrapper, func)

 

    # if there are no args, the decorator was used without params @decorator

    # otherwise, the decorator was called with params @decorator(arg)

    if len(args0) == 0:

        return outer_wrapper

    return outer_wrapper(args0[0])

parse_manifest 处理

def parse_manifest(runtime_config, write_perf_info, write, write_json):

   # 进行adapter 注册

    register_adapter(runtime_config, get_mp_context())

   # 使用注册的adapter 进行 macro context 处理

    adapter = get_adapter(runtime_config)

    adapter.set_macro_context_generator(generate_runtime_macro_context)

    # ManifestLoader 加载完整的manifest

    manifest = ManifestLoader.get_full_manifest(

        runtime_config,

        write_perf_info=write_perf_info,

    )

   # 写入manifest 到target 文件夹下

    if write and write_json:

        write_manifest(manifest, runtime_config.project_target_path)

        pm = plugins.get_plugin_manager(runtime_config.project_name)

        plugin_artifacts = pm.get_manifest_artifacts(manifest)

        for path, plugin_artifact in plugin_artifacts.items():

            plugin_artifact.write(path)

    return manifest

    )

说明

以上只是简单的说明manifest 参数是如何处理以及其他cli 参数的使用，详细解析里边比较复杂，后续会通过源码分析下实际的处理

参考资料

https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/cli/requires.py
https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/cli/main.py#L558
https://github.com/dbt-labs/dbt-core