dbt-checkpoint 源码结构简单说明

前边说过dbt-checkpoint 是基于dbt 的元数据解析，然后集合规则进行check，属于一个pre-commit 插件，以下简单说明下内部实现

配置

核心是 .pre-commit-hooks.yaml文件，一个标准的pre-commit 定义

• 内容

核心是id，name，entry，language，entry 实际上就是一个python entry_points 的 console_scripts

- id: check-column-desc-are-same

  name: Check column descriptions are same

  description: Check the models have same descriptions for same column names.

  entry: check-column-desc-are-same

  language: python

  files: '.*\.(yml|yaml)$'

- id: check-column-name-contract

  name: Check column name contract

  description: Check column name abides to contract.

  entry: check-column-name-contract

  language: python

  types_or: [sql, yaml]

- id: check-macro-has-description

  name: Check the macro has description

  description: Ensures that the macro has description in properties file.

  entry: check-macro-has-description

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process yaml and sql

- id: check-macro-arguments-have-desc

  name: Check the macro arguments have description

  description: Ensures that the macro has arguments with descriptions in properties file.

  entry: check-macro-arguments-have-desc

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process yaml and sql

- id: check-model-columns-have-desc

  name: Check the model columns have description

  description: Ensures that the model has columns with descriptions in properties file.

  entry: check-model-columns-have-desc

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process yaml and sql

- id: check-model-has-all-columns

  name: Check the model has all columns in properties file

  description: Ensures that all columns in database are specified in properties file.

  entry: check-model-has-all-columns

  language: python

  types_or: [sql, yaml]

- id: check-model-has-description

  name: Check the model has description

  description: Ensures that the model has description in properties file.

  entry: check-model-has-description

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process yaml and sql

- id: check-model-has-meta-keys

  name: Check the model has keys in the meta part

  description: Ensures that the model has a list of valid meta keys.

  entry: check-model-has-meta-keys

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process YAML and SQL

- id: check-model-has-labels-keys

  name: Check the model has keys in the labels part

  description: Ensures that the model has a list of valid labels keys.

  entry: check-model-has-labels-keys

  language: python

  types_or: [yaml, sql]

  require_serial: true # because we need to process YAML and SQL

- id: check-model-has-properties-file

  name: Check the model has properties file

  description: Ensures that the model has properties file (schema file).

  entry: check-model-has-properties-file

  language: python

  types_or: [sql, yaml]

- id: check-model-has-tests-by-group

  name: Check model; number of tests from group of tests.

  description: Ensures that the model has a number of tests from a group of tests.

  entry: check-model-has-tests-by-group

  language: python

  types_or: [sql, yaml]

- id: check-model-has-tests-by-name

  name: Check the model tests by a test name

  description: Ensures that the model has a number of tests of a certain name (e.g. data, unique).

  entry: check-model-has-tests-by-name

  language: python

  types_or: [sql, yaml]

- id: check-model-has-tests-by-type

  name: Check the model tests by a test type

  description: Ensures that the model has a number of tests of a certain type (data, schema).

  entry: check-model-has-tests-by-type

  language: python

  types_or: [sql, yaml]

- id: check-model-has-tests

  name: Check that model has tests

  description: Ensures that the model has a number of tests.

  entry: check-model-has-tests

  language: python

  types_or: [sql, yaml]

- id: check-model-name-contract

  name: Check model name contract

  description: Check model name abides to contract.

  entry: check-model-name-contract

  language: python

  types_or: [sql]

- id: check-model-parents-schema

  name: Check parent models/sources from certain schema

  entry: check-model-parents-schema

  language: python

  types_or: [yaml, sql]

- id: check-model-parents-database

  name: Check parent models/sources from certain database

  entry: check-model-parents-database

  language: python

  types_or: [sql, yaml]

- id: check-model-parents-and-childs

  name: Check the model has a parents/childs

  description: Ensures the model has a specific number (max/min) of parents or/and childs.

  entry: check-model-parents-and-childs

  language: python

  types_or: [sql, yaml]

- id: check-model-tags

  name: Check the model has valid tags

  description: Ensures that the model has only valid tags from the provided list.

  entry: check-model-tags

  language: python

  types_or: [sql, yaml]

- id: check-script-has-no-table-name

  name: Check the script has not table name

  description: Ensures that the script is using only source or ref macro to specify the table name.

  entry: check-script-has-no-table-name

  language: python

  types_or: [sql]

- id: check-script-ref-and-source

  name: Check the script has existing refs and sources

  description: Ensures that the script contains only existing sources or macros.

  entry: check-script-ref-and-source

  language: python

  types_or: [sql]

- id: check-script-semicolon

  name: Check the script does not contain a semicolon

  description: Ensure that the script does not have a semicolon at the end of the file.

  entry: check-script-semicolon

  language: python

  types_or: [sql]

- id: check-source-childs

  name: Check the source has max/min number of childs.

  description: Ensures the source has a specific number (max/min) of childs.

  entry: check-source-childs

  language: python

  types_or: [sql]

- id: check-source-columns-have-desc

  name: Check for source column descriptions

  description: Ensures that the source has columns with descriptions in the properties file.

  entry: check-source-columns-have-desc

  language: python

  types_or: [yaml]

- id: check-source-has-all-columns

  name: Check source has all columns in properties file

  description: Ensures that all columns in the database are specified in the properties file.

  entry: check-source-has-all-columns

  language: python

  types_or: [yaml]

- id: check-source-table-has-description

  name: Check the source table has description

  description: Ensures that the source table has description in properties file.

  entry: check-source-table-has-description

  language: python

  types_or: [yaml]

- id: check-source-has-freshness

  name: Check the source has the freshness

  description: Ensures that the source has freshness options.

  entry: check-source-has-freshness

  language: python

  types_or: [yaml]

- id: check-source-has-loader

  name: Check the source has loader option

  description: Ensures that the source has loader option.

  entry: check-source-has-loader

  language: python

  types_or: [yaml]

- id: check-source-has-meta-keys

  name: Check the source has keys in the meta part

  description: Ensures that the source has a list of valid meta keys.

  entry: check-source-has-meta-keys

  language: python

  types_or: [yaml]

- id: check-source-has-labels-keys

  name: Check the source has keys in the labels part

  description: Ensures that the source has a list of valid labels keys.

  entry: check-source-has-labels-keys

  language: python

  types_or: [yaml]

- id: check-source-has-tests-by-name

  name: Check the source tests by test name

  description: Ensures that the source has a number of tests of a certain name (e.g. data, unique).

  entry: check-source-has-tests-by-name

  language: python

  types_or: [yaml]

- id: check-source-has-tests-by-type

  name: Check the source tests by test type

  description: Ensures that the source has a number of tests of a certain type (data, schema).

  entry: check-source-has-tests-by-type

  language: python

  types_or: [yaml]

- id: check-source-has-tests-by-group

  name: Check the source tests by test group

  description: Ensures that the source has a number of tests of a certain group (unique, unique-combination-of-columns).

  entry: check-source-has-tests-by-group

  language: python

  types_or: [yaml]

- id: check-source-has-tests

  name: Check the source has tests

  description: Ensures that the source has a number of tests.

  entry: check-source-has-tests

  language: python

  types_or: [yaml]

- id: check-source-tags

  name: Check the source has valid tags

  description: Ensures that the source has only valid tags from the provided list.

  entry: check-source-tags

  language: python

  types_or: [yaml]

- id: dbt-clean

  name: dbt clean

  description: Deletes all folders specified in the clean-targets.

  entry: dbt-clean

  language: python

  pass_filenames: false

- id: dbt-compile

  name: dbt compile

  description: Generates executable SQL from source model, test, and analysis files.

  entry: dbt-compile

  language: python

  types_or: [sql]

  require_serial: true

- id: dbt-deps

  name: dbt deps

  description: Pulls the most recent version of the dependencies listed in your packages.yml.

  entry: dbt-deps

  language: python

  pass_filenames: false

- id: dbt-docs-generate

  name: dbt docs generate

  description: The command is responsible for generating your project's documentation website.

  entry: dbt-docs-generate

  language: python

  pass_filenames: false

- id: dbt-parse

  name: dbt parse

  description: Generates manifest.json from source model, test, and analysis files.

  entry: dbt-parse

  language: python

  types_or: [sql]

  require_serial: true

- id: dbt-run

  name: dbt run

  description: Executes compiled sql model files.

  entry: dbt-run

  language: python

  require_serial: true

  types_or: [sql]

- id: dbt-test

  name: dbt test

  description: Runs tests on data in deployed models.

  entry: dbt-test

  language: python

  require_serial: true

  types_or: [sql]

- id: generate-missing-sources

  name: Generate missing sources

  description: If any source is missing this hook tries to create it.

  entry: generate-missing-sources

  language: python

  types_or: [sql]

- id: generate-model-properties-file

  name: Generate model properties file

  description: Generate model properties file if does not exists.

  entry: generate-model-properties-file

  language: python

  types_or: [sql]

  args:

    [

      "--properties-file",

      "/Users/tomsejr/Documents/03-Workspace/Private/jaffle_shop/{database}/{schema}/{name}.yml",

    ]

  require_serial: true

- id: unify-column-description

  name: Unify column description

  description: Unify column descriptions across all models

  entry: unify-column-description

  language: python

  files: '.*\.(yml|yaml)$'

  require_serial: true

- id: replace-script-table-names

  name: Replace script table names

  description: Replace table names with source or ref macros in the script.

  entry: replace-script-table-names

  language: python

  types_or: [sql]

- id: remove-script-semicolon

  name: Remove script semicolon

  description: Remove semicolon at the end of the script.

  entry: remove-script-semicolon

  language: python

  types_or: [sql]

- id: check-model-materialization-by-childs

  name: Check the materialization of the model by childs

  description: Controls the materialization of the model by its number of childs.

  entry: check-model-materialization-by-childs

  language: python

  types_or: [sql]

- id: check-exposure-has-meta-keys

  name: Check the exposure has keys in the meta part

  description: Ensures that the exposure has a list of valid meta keys.

  entry: check-exposure-has-meta-keys

  language: python

  types_or: [yaml]

- id: check-macro-has-meta-keys

  name: Check the macro has keys in the meta part

  description: Ensures that the macro has a list of valid meta keys.

  entry: check-macro-has-meta-keys

  language: python

  types_or: [yaml]

- id: check-seed-has-meta-keys

  name: Check the seed has keys in the meta part

  description: Ensures that the seed has a list of valid meta keys.

  entry: check-seed-has-meta-keys

  language: python

  types_or: [yaml]

- id: check-snapshot-has-meta-keys

  name: Check the snapshot has keys in the meta part

  description: Ensures that the snapshot has a list of valid meta keys.

  entry: check-snapshot-has-meta-keys

  language: python

  types_or: [sql, yaml]

- id: check-test-has-meta-keys

  name: Check the test has keys in the meta part

  description: Ensures that the test has a list of valid meta keys.

  entry: check-test-has-meta-keys

  language: python

  types_or: [sql]

entry 实现

一个简单例子，实际就是解析元数据，基于规则判断,为了方便处理，开发了一个工具模块utils.py 定义了类型以及元数据解析处理

• check_macro_has_description.py

import argparse

import os

import time

from typing import Any, Dict, Optional, Sequence

 

from dbt_checkpoint.tracking import dbtCheckpointTracking

from dbt_checkpoint.utils import (

    JsonOpenError,

    add_default_args,

    get_dbt_manifest,

    get_filenames,

    get_macro_schemas,

    get_macro_sqls,

    get_macros,

    get_missing_file_paths,

    red,

)

 

#  基于元数据的规则处理

def has_description(

    paths: Sequence[str], manifest: Dict[str, Any], exclude_pattern: str

) -> Dict[str, Any]:

    paths = get_missing_file_paths(paths, manifest, exclude_pattern=exclude_pattern)

    status_code = 0

    ymls = get_filenames(paths, [".yml", ".yaml"])

    sqls = get_macro_sqls(paths, manifest)

    filenames = set(sqls.keys())

 

    # get manifest macros that pre-commit found as changed

    macros = get_macros(manifest, filenames)

    # if user added schema but did not rerun the macro

    schemas = get_macro_schemas(list(ymls.values()), filenames)

    # convert to sets

    in_macros = {macro.filename for macro in macros if macro.macro.get("description")}

    in_schemas = {

        schema.macro_name for schema in schemas if schema.schema.get("description")

    }

    missing = filenames.difference(in_macros, in_schemas)

 

    for macro in missing:

        status_code = 1

        print(

            f"{red(sqls.get(macro))}: "

            f"does not have defined description or properties file is missing.",

        )

    return {"status_code": status_code}

 

 

def main(argv: Optional[Sequence[str]] = None) -> int:

    parser = argparse.ArgumentParser()

    add_default_args(parser)

 

    args = parser.parse_args(argv)

 

    try:

        manifest = get_dbt_manifest(args)

    except JsonOpenError as e:

        print(f"Unable to load manifest file ({e})")

        return 1

 

    start_time = time.time()

    hook_properties = has_description(

        paths=args.filenames, manifest=manifest, exclude_pattern=args.exclude

    )

    end_time = time.time()

    script_args = vars(args)

 

    tracker = dbtCheckpointTracking(script_args=script_args)

    tracker.track_hook_event(

        event_name="Hook Executed",

        manifest=manifest,

        event_properties={

            "hook_name": os.path.basename(__file__),

            "description": "Check the macro has description.",

            "status": hook_properties.get("status_code"),

            "execution_time": end_time - start_time,

            "is_pytest": script_args.get("is_test"),

        },

    )

 

    return hook_properties.get("status_code")

 

 

if __name__ == "__main__":

    exit(main())

• entry_points 的 console_scripts 定义

[options.entry_points]

console_scripts =

    check-column-desc-are-same = dbt_checkpoint.check_column_desc_are_same:main

    check-column-name-contract = dbt_checkpoint.check_column_name_contract:main

    check-macro-has-description = dbt_checkpoint.check_macro_has_description:main

    check-macro-arguments-have-desc = dbt_checkpoint.check_macro_arguments_have_desc:main

    check-model-columns-have-desc = dbt_checkpoint.check_model_columns_have_desc:main

    check-model-has-all-columns = dbt_checkpoint.check_model_has_all_columns:main

    check-model-has-description = dbt_checkpoint.check_model_has_description:main

    check-model-has-meta-keys = dbt_checkpoint.check_model_has_meta_keys:main

    check-model-has-labels-keys = dbt_checkpoint.check_model_has_labels_keys:main

    check-model-has-properties-file = dbt_checkpoint.check_model_has_properties_file:main

    check-model-has-tests-by-name = dbt_checkpoint.check_model_has_tests_by_name:main

    check-model-has-tests-by-type = dbt_checkpoint.check_model_has_tests_by_type:main

    check-model-has-tests-by-group = dbt_checkpoint.check_model_has_tests_by_group:main

    check-model-has-tests = dbt_checkpoint.check_model_has_tests:main

    check-model-name-contract = dbt_checkpoint.check_model_name_contract:main

    check-model-parents-and-childs = dbt_checkpoint.check_model_parents_and_childs:main

    check-model-parents-database = dbt_checkpoint.check_model_parents_database:main

    check-model-parents-schema = dbt_checkpoint.check_model_parents_schema:main

    check-model-tags = dbt_checkpoint.check_model_tags:main

    check-script-has-no-table-name = dbt_checkpoint.check_script_has_no_table_name:main

    check-script-ref-and-source = dbt_checkpoint.check_script_ref_and_source:main

    check-script-semicolon = dbt_checkpoint.check_script_semicolon:main

    check-source-childs = dbt_checkpoint.check_source_childs:main

    check-source-columns-have-desc = dbt_checkpoint.check_source_columns_have_desc:main

    check-source-has-all-columns = dbt_checkpoint.check_source_has_all_columns:main

    check-source-table-has-description = dbt_checkpoint.check_source_table_has_description:main

    check-source-has-freshness = dbt_checkpoint.check_source_has_freshness:main

    check-source-has-loader = dbt_checkpoint.check_source_has_loader:main

    check-source-has-meta-keys = dbt_checkpoint.check_source_has_meta_keys:main

    check-source-has-labels-keys = dbt_checkpoint.check_source_has_labels_keys:main

    check-source-has-tests-by-group = dbt_checkpoint.check_source_has_tests_by_group:main

    check-source-has-tests-by-name = dbt_checkpoint.check_source_has_tests_by_name:main

    check-source-has-tests-by-type = dbt_checkpoint.check_source_has_tests_by_type:main

    check-source-has-tests = dbt_checkpoint.check_source_has_tests:main

    check-source-tags = dbt_checkpoint.check_source_tags:main

    check-model-materialization-by-childs = dbt_checkpoint.check_model_materialization_by_childs:main

    dbt-clean = dbt_checkpoint.dbt_clean:main

    dbt-compile = dbt_checkpoint.dbt_compile:main

    dbt-deps = dbt_checkpoint.dbt_deps:main

    dbt-docs-generate = dbt_checkpoint.dbt_docs_generate:main

    dbt-parse = dbt_checkpoint.dbt_parse:main

    dbt-run = dbt_checkpoint.dbt_run:main

    dbt-test = dbt_checkpoint.dbt_test:main

    generate-missing-sources = dbt_checkpoint.generate_missing_sources:main

    generate-model-properties-file = dbt_checkpoint.generate_model_properties_file:main

    unify-column-description = dbt_checkpoint.unify_column_description:main

    replace-script-table-names = dbt_checkpoint.replace_script_table_names:main

    remove-script-semicolon = dbt_checkpoint.remove_script_semicolon:main

    check-snapshot-has-meta-keys = dbt_checkpoint.check_snapshot_has_meta_keys:main

    check-exposure-has-meta-keys = dbt_checkpoint.check_exposure_has_meta_keys:main

    check-macro-has-meta-keys = dbt_checkpoint.check_macro_has_meta_keys:main

    check-seed-has-meta-keys = dbt_checkpoint.check_seed_has_meta_keys:main

    check-test-has-meta-keys = dbt_checkpoint.check_test_has_meta_keys:main

使用

对于使用实际上就是标准的pre-commit 玩法

• 定义.pre-commit-config.yaml

里边会包含hooks 定义

repos:

- repo: https://github.com/dbt-checkpoint/dbt-checkpoint

  rev: v1.2.1

  hooks:

  - id: dbt-parse

  - id: dbt-docs-generate

    args: ["--cmd-flags", "++no-compile"]

  - id: check-script-semicolon

  - id: check-script-has-no-table-name

  - id: check-model-has-all-columns

    name: Check columns - core

    files: ^models/core

  - id: check-model-has-all-columns

    name: Check columns - mart

    files: ^models/mart

  - id: check-model-columns-have-desc

    files: ^models/mart

• 运行

可以安装为git 的pre-commit （pre-commit install）在git commit 的时候check，也可以直接运行pre-commit run --all-files

说明

dbt-checkpoint 核心是解析dbt元数据，但是也提供了dbt core 一个cli 命令的包装，比如run，compile，deps，clean，test，docs generate，parse 值得试用下

参考资料

https://github.com/dbt-checkpoint/dbt-checkpoint
https://pre-commit.com/hooks.html