dremio dbt ref 内部处理简单说明

dremio dbt 内部对于ref 进行了重写，可以实现一些灵活的扩展

参考处理

• ref 参考实现 dbt/include/dremio/macros/builtins/builtins.sql
  对于ref 进行了一些调整，可以进行一些自定义操作，核心是对于模型格式化的定义

{%- macro ref(model_name) -%}

  {%- set relation = builtins.ref(model_name) -%}

  {%- if execute -%}

    {%- set model = graph.nodes.values() | selectattr("name", "equalto", model_name) | list | first -%}

    {%- if model.config.materialized == 'reflection' -%}

      {% do exceptions.CompilationError("Reflections cannot be ref()erenced (" ~ relation ~ ")") %}

    {%- endif -%}

    {%- set format = model.config.format if

      model.config.materialized not in ['view', 'reflection']

      and model.config.format is defined

      else none -%}

    {%- set format_clause = format_clause_from_node(model.config) if format is not none else none -%}

    {%- set relation2 = api.Relation.create(database=relation.database, schema=relation.schema, identifier=relation.identifier, format=format, format_clause=format_clause) -%}

    {{ return (relation2) }}

  {%- else -%}

    {{ return (relation) }}

  {%- endif -%}

{%- endmacro -%} 

参考执行效果

02:18:11  relation origin ref: "dbtv4"."dbtv4"."biz"."my_mydemoapp"

02:18:11  parse  relation: "dbtv4"."dbtv4"."biz"."my_mydemoapp" view

02:18:11  mydemoapp "dbtv4"."dbtv4"."biz"."my_mydemoapp" mydemoapp 

02:18:11  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.

There are 2 unused configuration paths:

- models.dremio_app

- seeds.dremio.app

02:18:11  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 351 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics, 0 groups

02:18:11  

02:18:12  Concurrency: 3 threads (target='dev')

02:18:12  

02:18:12  1 of 1 START sql table model s3.dbtv4.my_mydemoapp ............................. [RUN]

02:18:12  relation origin ref: "s3"."dbtv4"."app"

02:18:12  exec relation : "s3"."dbtv4"."app"

02:18:12  exec relation2: "s3"."dbtv4"."app"

02:18:12  mydemoapp "s3"."dbtv4"."my_mydemoapp" mydemoapp table

02:18:13  Creating or replacing view "dbtv4"."dbtv4"."biz"."my_mydemoapp"

02:18:14  1 of 1 OK created sql table model s3.dbtv4.my_mydemoapp ........................ [3 in 1.83s]

02:18:14  

02:18:14  Finished running 1 table model in 0 hours 0 minutes and 2.64 seconds (2.64s).

说明

对于ref 的重写在一些场景下还是很有用的，比如自定义relation的信息实现不同配置查询生成不同的模型信息

参考资料

https://docs.getdbt.com/reference/dbt-jinja-functions/execute
https://docs.getdbt.com/reference/dbt-jinja-functions/ref
dbt/include/dremio/macros/materializations/view/create_or_replace_view.sql
https://docs.getdbt.com/reference/dbt-jinja-functions/target