prefect - The easiest way to automate your data
prefect
https://github.com/PrefectHQ/prefect
https://docs.prefect.io/
Prefect
We've rebuilt data engineering for the data science era.
Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize
Tasks
intoFlows
, and Prefect takes care of the rest.
Demo code
Prefect's Pythonic API should feel familiar for newcomers. Mark functions as tasks and call them on each other to build up a flow.
from prefect import task, Flow, Parameter @task(log_stdout=True) def say_hello(name): print("Hello, {}!".format(name)) with Flow("My First Flow") as flow: name = Parameter('name') say_hello(name) flow.run(name='world') # "Hello, world!" flow.run(name='Marvin') # "Hello, Marvin!"
安装
https://docs.prefect.io/core/getting_started/installation.html
https://docs.docker.com/compose/install/
发布
https://docs.prefect.io/orchestration/concepts/flows.html#registration
https://docs.prefect.io/api/latest/cli/create.html
prefect create project "my project"
prefect register --project "my project" --path test.py