dbt config macro 简单说明

dbt 不少资源类型都支持config macro 的使用，可以进行灵活的配置管理

参考使用

  • 配置
    模型的执行物化策略
{{ config(
    materialized="<materialization_name>",
    sql_header="<string>"
) }}

快照的

{{ config(
    target_schema="<string>",
    target_database="<string>",
    unique_key="<column_name_or_expression>",
    strategy="timestamp" | "check",
    updated_at="<column_name>",
    check_cols=["<column_name>"] | "all"
) }}
  • 获取config
{% materialization incremental, default -%}
  {%- set unique_key = config.get('unique_key') -%}
  ...
  • 提供的方法
    get，require，

内部处理

实际上就是一个类，实现了一些方法

  • 参考配置
class Config(Protocol):
    def __init__(self, model, context_config: Optional[ContextConfig]):
        ...
 
# 这个是解析的，当然还有不少其他的实现
# Implementation of "config(..)" calls in models
class ParseConfigObject(Config):
    def __init__(self, model, context_config: Optional[ContextConfig]):
        self.model = model
        self.context_config = context_config
 
    def _transform_config(self, config):
        for oldkey in ("pre_hook", "post_hook"):
            if oldkey in config:
                newkey = oldkey.replace("_", "-")
                if newkey in config:
                    raise ConflictingConfigKeysError(oldkey, newkey, node=self.model)
                config[newkey] = config.pop(oldkey)
        return config
 
    def __call__(self, *args, **kwargs):
        if len(args) == 1 and len(kwargs) == 0:
            opts = args[0]
        elif len(args) == 0 and len(kwargs) > 0:
            opts = kwargs
        else:
            raise InlineModelConfigError(node=self.model)
 
        opts = self._transform_config(opts)
 
        # it's ok to have a parse context with no context config, but you must
        # not call it!
        if self.context_config is None:
            raise DbtRuntimeError("At parse time, did not receive a context config")
        self.context_config.add_config_call(opts)
        return ""
 
    def set(self, name, value):
        return self.__call__({name: value})
 
    def require(self, name, validator=None):
        return ""
 
    def get(self, name, default=None, validator=None):
        return ""
 
    def persist_relation_docs(self) -> bool:
        return False
 
    def persist_column_docs(self) -> bool:
        return False
  • 上下文处理
    render_with_context 会使用_context_for ，_context_for会调用generate_parser_model_context 会使用到config
def generate_parser_model_context(
    model: ManifestNode,
    config: RuntimeConfig,
    manifest: Manifest,
    context_config: ContextConfig,
) -> Dict[str, Any]:
    # The __init__ method of ModelContext also initializes
    # a ManifestContext object which creates a MacroNamespaceBuilder
    # which adds every macro in the Manifest.
    ctx = ModelContext(model, config, manifest, ParseProvider(), context_config)
    # The 'to_dict' method in ManifestContext moves all of the macro names
    # in the macro 'namespace' up to top level keys
    return ctx.to_dict()

说明

以上是关于dbt config的简单说明，对于详细使用可以参考官方文档

参考资料

https://docs.getdbt.com/reference/data-test-configs
https://docs.getdbt.com/reference/dbt-jinja-functions/config

posted on 2024-05-12 06:51  荣锋亮  阅读(21)  评论(0)  编辑  收藏  举报

导航