dbt adapter get_incremental_strategy_macro 简单说明

dbt adapter 的get_incremental_strategy_macro 核心获取增量策略的macro 名称，可以参考我以前写的一些

使用

目前就是应用在dbt 的增量场景中

• 增量参考使用

{% set incremental_strategy = config.get('incremental_strategy') or 'default' %}

{% set incremental_predicates = config.get('predicates', none) or config.get('incremental_predicates', none) %}

{% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}

{% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': temp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'incremental_predicates': incremental_predicates }) %}

{% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}

参考实现

属于adapter 的一个方法，核心是基于context，结合策略名称查找对应的可执行macro

• 内部处理

@available.parse_none

def get_incremental_strategy_macro(self, model_context, strategy: str):

    # Construct macro_name from strategy name

    if strategy is None:

        strategy = "default"

 

    # validate strategies for this adapter

   # valid_incremental_strategies 不同adapter 的实现是不同的，比如默认的只有append，但是pg 就比较多来了

    valid_strategies = self.valid_incremental_strategies()

    valid_strategies.append("default")

    builtin_strategies = self.builtin_incremental_strategies()

    if strategy in builtin_strategies and strategy not in valid_strategies:

        raise DbtRuntimeError(

            f"The incremental strategy '{strategy}' is not valid for this adapter"

        )

 

    strategy = strategy.replace("+", "_")

    macro_name = f"get_incremental_{strategy}_sql"

    # The model_context should have callable objects for all macros

    if macro_name not in model_context:

        raise DbtRuntimeError(

            'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(

                macro_name, self.config.project_name

            )

        )

 

    # This returns a callable macro

    return model_context[macro_name]

一些实现

def valid_incremental_strategies(self):

    """The set of standard builtin strategies which this adapter supports out-of-the-box.

    Not used to validate custom strategies defined by end users.

    """

    return ["append"]

 

def builtin_incremental_strategies(self):

    return ["append", "delete+insert", "merge", "insert_overwrite"]

说明

dbt get_incremental_strategy_macro 核心是提供了一个增量macro 的策略macro 查找处理，了解内部的机制还是比较有用的

参考资料

dbt/adapters/base/impl.py
https://www.cnblogs.com/rongfengliang/p/18108984