修改官方cubestore 支持minio s3

支持s3是cubestore 一个很不错的特性，可以提高系统的扩展性

参考修改

主要是添加endpoint 以及配置使用path格式请求
s3.rs

 
impl S3RemoteFs {
    pub fn new(
        dir: PathBuf,
        region: String,
        endpoint:String,
        bucket_name: String,
        sub_path: Option<String>,
    ) -> Result<Arc<Self>, CubeError> {
        let credentials = Credentials::new(
            env::var("CUBESTORE_AWS_ACCESS_KEY_ID").as_deref().ok(),
            env::var("CUBESTORE_AWS_SECRET_ACCESS_KEY").as_deref().ok(),
            None,
            None,
            None,
        )?;
        let bucket = Bucket::new_with_path_style(&bucket_name, Region::Custom {
            endpoint:endpoint,
            region:region,
        }, credentials)?;
        Ok(Arc::new(Self {
            dir,
            bucket,
            sub_path,
            delete_mut: Mutex::new(()),
        }))
    }
}

config/mod.rs

#[derive(Debug, Clone)]
pub enum FileStoreProvider {
    Local,
    Filesystem {
        remote_dir: Option<PathBuf>,
    },
    S3 {
        region: String,
        endpoint:String,
        bucket_name: String,
        sub_path: Option<String>,
    },
    GCS {
        bucket_name: String,
        sub_path: Option<String>,
    },
}
  • 添加环境变量支持
    摘取了部分，详细参考源码
 
store_provider: {
        if let Ok(bucket_name) = env::var("CUBESTORE_S3_BUCKET") {
            FileStoreProvider::S3 {
                bucket_name,
                region: env::var("CUBESTORE_S3_REGION").unwrap(),
                endpoint: env::var("CUBESTORE_S3_ENDPOINT").unwrap(),
                sub_path: env::var("CUBESTORE_S3_SUB_PATH").ok(),
            }
        } else if let Ok(bucket_name) = env::var("CUBESTORE_GCS_BUCKET") {
            FileStoreProvider::GCS {
                bucket_name,
                sub_path: env::var("CUBESTORE_GCS_SUB_PATH").ok(),
            }
        } else if let Ok(remote_dir) = env::var("CUBESTORE_REMOTE_DIR") {
            FileStoreProvider::Filesystem {
                remote_dir: Some(PathBuf::from(remote_dir)),
            }
        } else {
            FileStoreProvider::Filesystem { remote_dir: None }
        }
    },

运行试用

  • 参考配置
export CUBESTORE_LOG_LEVEL=debug
export CUBESTORE_SERVER_NAME=localhost:9999
export CUBESTORE_META_PORT=9999
export CUBESTORE_S3_BUCKET=test
export CUBESTORE_S3_ENDPOINT=http://localhost:9000
export CUBESTORE_S3_REGION=us-east-1
export CUBESTORE_AWS_ACCESS_KEY_ID=dalongrong
export CUBESTORE_AWS_SECRET_ACCESS_KEY=dalongrong
 ./target/debug/cubestored
  • 效果

 

 

说明

以上是一个简单的使用，关于集群以及docker镜像，后期构建好会发布到官方仓库

参考资料

https://github.com/rongfengliang/mycubejs
https://github.com/durch/rust-s3/blob/master/s3/bin/simple_crud.rs
https://cube.dev/docs/caching/using-pre-aggregations#running-in-production

posted on 2021-05-02 00:31  荣锋亮  阅读(135)  评论(0)  编辑  收藏  举报

导航