pandas 加载minio 文件数据

就是一个简单记录，基于s3 进行文件存储还是比较方便的

环境准备

• docker-compose.yaml

version: '3'

services:

  minio:

    image: minio/minio

    ports:

      - "9002:9000"

      - "19001:19001"

    environment:

      MINIO_ACCESS_KEY: minio

      MINIO_SECRET_KEY: minio123

    command: server --console-address :19001 --quiet /data

• requirements.txt
  主要是依赖，方便通过venv 安装

 

pandas

s3fs

fsspec

pyarrow

读取&效果

• s3app.py

import pandas as pd

parquet_file = 's3://demo/file2.parquet'

storage_conf = {'endpoint_url':'http://localhost:9002',"key":"minio","secret":"minio123"}

result = pd.read_parquet(parquet_file, engine='pyarrow',storage_options=storage_conf)

print(result.head())

• 效果

说明

pandas 对于s3 的读取依赖了fsspec，同时实际读取需要安装对应的s3 实现

参考资料

https://s3fs.readthedocs.io/en/latest/
https://filesystem-spec.readthedocs.io/en/latest/
https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html