dlt s3 集成试用

属于一个简单试用，dlt 支持destinations 为filesystem，当然也支持source 为filesystem，内部处理是使用了s3fs

环境准备

这个比较简单，推荐基于venv

pip install dlt[filesystem]

通过docker 部署，同时可以需要创建demo bucket

version: "3"

services:

  minio:

    image: minio/minio

    ports:

      - "9000:9000"

      - "19001:19001"

    environment:

      MINIO_ACCESS_KEY: minio

      MINIO_SECRET_KEY: minio123

    command: server --console-address :19001 --quiet /data

配置文件系统的key

可以直接在项目的.dlt 文件夹下.dlt/secrets.toml

[destination.filesystem]

bucket_url = "s3://demo" # replace with your bucket name,

[destination.filesystem.credentials]

aws_access_key_id = "minio" # copy the access key here

aws_secret_access_key = "minio123" # copy the secret access key here

endpoint_url = "http://localhost:9000" # copy your endpoint URL here

数据处理

s3.py

import dlt
 
data = [{'id': 1, 'name': 'John','age':111}, {'id': 2, 'name': 'Jane'}]
 
pipeline = dlt.pipeline(

    pipeline_name="dalong",

    destination='filesystem',

    dataset_name='postgres_data'

)
 
load_info = pipeline.run(

    data,

    write_disposition="merge",

    table_name='postgres_data',

    # 数据存储为parquet 格式

    loader_file_format='parquet',

)
 
print(load_info)