dbt adapter get_incremental_strategy_macro 简单说明
dbt adapter 的get_incremental_strategy_macro 核心获取增量策略的macro 名称,可以参考我以前写的一些
使用
目前就是应用在dbt 的增量场景中
- 增量参考使用
{% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
{% set incremental_predicates = config.get('predicates', none) or config.get('incremental_predicates', none) %}
{% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}
{% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': temp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'incremental_predicates': incremental_predicates }) %}
{% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}
参考实现
属于adapter 的一个方法,核心是基于context,结合策略名称查找对应的可执行macro
- 内部处理
@available.parse_none
def get_incremental_strategy_macro(self, model_context, strategy: str):
# Construct macro_name from strategy name
if strategy is None:
strategy = "default"
# validate strategies for this adapter
# valid_incremental_strategies 不同adapter 的实现是不同的,比如默认的只有append,但是pg 就比较多来了
valid_strategies = self.valid_incremental_strategies()
valid_strategies.append("default")
builtin_strategies = self.builtin_incremental_strategies()
if strategy in builtin_strategies and strategy not in valid_strategies:
raise DbtRuntimeError(
f"The incremental strategy '{strategy}' is not valid for this adapter"
)
strategy = strategy.replace("+", "_")
macro_name = f"get_incremental_{strategy}_sql"
# The model_context should have callable objects for all macros
if macro_name not in model_context:
raise DbtRuntimeError(
'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(
macro_name, self.config.project_name
)
)
# This returns a callable macro
return model_context[macro_name]
一些实现
def valid_incremental_strategies(self):
"""The set of standard builtin strategies which this adapter supports out-of-the-box.
Not used to validate custom strategies defined by end users.
"""
return ["append"]
def builtin_incremental_strategies(self):
return ["append", "delete+insert", "merge", "insert_overwrite"]
说明
dbt get_incremental_strategy_macro 核心是提供了一个增量macro 的策略macro 查找处理,了解内部的机制还是比较有用的
参考资料
dbt/adapters/base/impl.py
https://www.cnblogs.com/rongfengliang/p/18108984