dbt click包执行上下文manifest 处理简单说明
dbt 的cli 执行是基于了click 者pip 包，同时为了方便执行（比如依赖的参数），dbt 核心比较重要的是manifest
这个参数是通过context 传递的
run 参考task 处理
- 参考代码
core dbt/cli/main.py
@cli.command("run")
@click.pass_context
@global_flags
@p.exclude
@p.full_refresh
@p.profile
@p.profiles_dir
@p.project_dir
@p.empty
@p.select
@p.selector
@p.target
@p.target_path
@p.threads
@p.vars
@requires.postflight
@requires.preflight
@requires.profile
@requires.project
@requires.runtime_config
@requires.manifest
def run(ctx, **kwargs):
"""Compile SQL and execute against the current target database."""
task = RunTask(
ctx.obj["flags"],
ctx.obj["runtime_config"],
ctx.obj["manifest"],
)
- 简单说明
cli.command 以及 click.pass_context 就是标准的click 配置p 是配置参数相关的，requires 是必须项，db基于了装饰器进行处理
@requires.manifest 是一个比较重要的东西dbt 好多命令都是需要此参数的， ctx.obj["manifest"] 此数据是通过装饰器处理的
manifest 参考处理
- core requires.py
def manifest(*args0, write=True, write_perf_info=False):
"""A decorator used by click command functions for generating a manifest
given a profile, project, and runtime config. This also registers the adapter
from the runtime config and conditionally writes the manifest to disk.
"""
def outer_wrapper(func):
def wrapper(*args, **kwargs):
ctx = args[0]
assert isinstance(ctx, Context)
req_strs = ["profile", "project", "runtime_config"]
reqs = [ctx.obj.get(dep) for dep in req_strs]
if None in reqs:
raise DbtProjectError("profile, project, and runtime_config required for manifest")
runtime_config = ctx.obj["runtime_config"]
# a manifest has already been set on the context, so don't overwrite it
if ctx.obj.get("manifest") is None:
# 如果数据为空，会进行解析生成（会结合runtime 配置信息）
ctx.obj["manifest"] = parse_manifest(
runtime_config, write_perf_info, write, ctx.obj["flags"].write_json
)
else:
# 已经包含了需要的信息，会进行adapter 的注册，以及macro 的解析处理
register_adapter(runtime_config, get_mp_context())
adapter = get_adapter(runtime_config)
adapter.set_macro_context_generator(generate_runtime_macro_context)
adapter.set_macro_resolver(ctx.obj["manifest"])
query_header_context = generate_query_header_context(
adapter.config, ctx.obj["manifest"]
)
adapter.connections.set_query_header(query_header_context)
return func(*args, **kwargs)
return update_wrapper(wrapper, func)
# if there are no args, the decorator was used without params @decorator
# otherwise, the decorator was called with params @decorator(arg)
if len(args0) == 0:
return outer_wrapper
return outer_wrapper(args0[0])
parse_manifest 处理
def parse_manifest(runtime_config, write_perf_info, write, write_json):
# 进行adapter 注册
register_adapter(runtime_config, get_mp_context())
# 使用注册的adapter 进行 macro context 处理
adapter = get_adapter(runtime_config)
adapter.set_macro_context_generator(generate_runtime_macro_context)
# ManifestLoader 加载完整的manifest
manifest = ManifestLoader.get_full_manifest(
runtime_config,
write_perf_info=write_perf_info,
)
# 写入manifest 到target 文件夹下
if write and write_json:
write_manifest(manifest, runtime_config.project_target_path)
pm = plugins.get_plugin_manager(runtime_config.project_name)
plugin_artifacts = pm.get_manifest_artifacts(manifest)
for path, plugin_artifact in plugin_artifacts.items():
plugin_artifact.write(path)
return manifest
)
说明
以上只是简单的说明manifest 参数是如何处理以及其他cli 参数的使用，详细解析里边比较复杂，后续会通过源码分析下实际的处理
参考资料
https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/cli/requires.py
https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/cli/main.py#L558
https://github.com/dbt-labs/dbt-core
