修改官方cubestore 支持minio s3

支持s3是cubestore 一个很不错的特性，可以提高系统的扩展性

参考修改

主要是添加endpoint 以及配置使用path格式请求
s3.rs

 

impl S3RemoteFs {

    pub fn new(

        dir: PathBuf,

        region: String,

        endpoint:String,

        bucket_name: String,

        sub_path: Option<String>,

    ) -> Result<Arc<Self>, CubeError> {

        let credentials = Credentials::new(

            env::var("CUBESTORE_AWS_ACCESS_KEY_ID").as_deref().ok(),

            env::var("CUBESTORE_AWS_SECRET_ACCESS_KEY").as_deref().ok(),

            None,

            None,

            None,

        )?;

        let bucket = Bucket::new_with_path_style(&bucket_name, Region::Custom {

            endpoint:endpoint,

            region:region,

        }, credentials)?;

        Ok(Arc::new(Self {

            dir,

            bucket,

            sub_path,

            delete_mut: Mutex::new(()),

        }))

    }

}

config/mod.rs

#[derive(Debug, Clone)]

pub enum FileStoreProvider {

    Local,

    Filesystem {

        remote_dir: Option<PathBuf>,

    },

    S3 {

        region: String,

        endpoint:String,

        bucket_name: String,

        sub_path: Option<String>,

    },

    GCS {

        bucket_name: String,

        sub_path: Option<String>,

    },

}

• 添加环境变量支持
  摘取了部分，详细参考源码

 

store_provider: {

        if let Ok(bucket_name) = env::var("CUBESTORE_S3_BUCKET") {

            FileStoreProvider::S3 {

                bucket_name,

                region: env::var("CUBESTORE_S3_REGION").unwrap(),

                endpoint: env::var("CUBESTORE_S3_ENDPOINT").unwrap(),

                sub_path: env::var("CUBESTORE_S3_SUB_PATH").ok(),

            }

        } else if let Ok(bucket_name) = env::var("CUBESTORE_GCS_BUCKET") {

            FileStoreProvider::GCS {

                bucket_name,

                sub_path: env::var("CUBESTORE_GCS_SUB_PATH").ok(),

            }

        } else if let Ok(remote_dir) = env::var("CUBESTORE_REMOTE_DIR") {

            FileStoreProvider::Filesystem {

                remote_dir: Some(PathBuf::from(remote_dir)),

            }

        } else {

            FileStoreProvider::Filesystem { remote_dir: None }

        }

    },

运行试用

• 参考配置

export CUBESTORE_LOG_LEVEL=debug

export CUBESTORE_SERVER_NAME=localhost:9999

export CUBESTORE_META_PORT=9999

export CUBESTORE_S3_BUCKET=test

export CUBESTORE_S3_ENDPOINT=http://localhost:9000

export CUBESTORE_S3_REGION=us-east-1

export CUBESTORE_AWS_ACCESS_KEY_ID=dalongrong

export CUBESTORE_AWS_SECRET_ACCESS_KEY=dalongrong

 ./target/debug/cubestored 

• 效果

 

 

说明

以上是一个简单的使用，关于集群以及docker镜像，后期构建好会发布到官方仓库

参考资料

https://github.com/rongfengliang/mycubejs
https://github.com/durch/rust-s3/blob/master/s3/bin/simple_crud.rs
https://cube.dev/docs/caching/using-pre-aggregations#running-in-production