zenoh s3 存储插件简单说明
以前简单介绍过关于zenoh 存储插件的接口定义，以下简单说明下s3实现的处理以及使用
使用
zenoh 对于存储插件的处理统一是通过存储管理，之后是配置具体的存储插件时间
- 参考配置
"plugins": {
// configuration of "storage-manager" plugin:
"storage_manager": {
"volumes": {
// configuration of a "fs" volume (the "zenoh_backend_fs" backend library will be loaded at startup)
"fs": {},
"s3": {
"region": "eu-west-1",
"url": "http://172.16.238.10:9000"
}
},
"storages": {
"s3_test": {
// the key expression this storage will subscribes to
"key_expr": "demo/**",
// this prefix will be stripped from the received key when converting to file path
// this argument is optional.
// "strip_prefix": "demo/example",
"volume": {
"id": "s3",
// the key/values will be stored as files within this directory (relative to ${ZENOH_BACKEND_FS_ROOT})
"bucket": "zenoh-bucket-3",
"reuse_bucket": true,
"private": {
// Credentials for interacting with the S3 bucket
"access_key": "minio",
"secret_key": "minio123"
}
}
}
}
}
}
内部处理
以前简单说过存储插件内部机制，对于s3的实现机制上也是一样的，支持是基于s3进行数据存储
- 存储插件接口实现
#[async_trait]
impl Storage for S3Storage {
fn get_admin_status(&self) -> serde_json::Value {
self.config.admin_status.to_owned()
}
/// Function to retrieve the sample associated with a single key.
async fn get(
&mut self,
key: Option<OwnedKeyExpr>,
_parameters: &str,
) -> ZResult<Vec<StoredData>> {
let key = key.map_or_else(|| OwnedKeyExpr::from_str(NONE_KEY), Ok)?;
tracing::debug!("GET called on client {}. Key: '{}'", self.client, key);
let s3_key = S3Key::from_key_expr(self.config.path_prefix.as_ref(), key.to_owned())?;
// 对于时间信息的存储基于了对象存储的emetadata
let get_result = self.get_stored_value(&s3_key.into()).await?;
if let Some((timestamp, value)) = get_result {
let stored_data = StoredData { value, timestamp };
Ok(vec![stored_data])
} else {
Ok(vec![])
}
}
/// Function called for each incoming data ([`Sample`]) to be stored in this storage.
async fn put(
&mut self,
key: Option<OwnedKeyExpr>,
value: Value,
timestamp: Timestamp,
) -> ZResult<StorageInsertionResult> {
let key = key.map_or_else(|| OwnedKeyExpr::from_str(NONE_KEY), Ok)?;
tracing::debug!("Put called on client {}. Key: '{}'", self.client, key);
let s3_key = S3Key::from_key_expr(self.config.path_prefix.as_ref(), key)
.map_or_else(|err| Err(zerror!("Error getting s3 key: {}", err)), Ok)?;
// 对于时间信息的存储基于了对象存储的emetadata
if !self.config.is_read_only {
let mut metadata: HashMap<String, String> = HashMap::new();
metadata.insert(TIMESTAMP_METADATA_KEY.to_string(), timestamp.to_string());
let key: String = s3_key.into();
let client = self.client.clone();
await_task!(client.put_object(key, value, Some(metadata)).await,)
.map_err(|e| zerror!("Put operation failed: {e}"))?;
Ok(StorageInsertionResult::Inserted)
} else {
tracing::warn!("Received PUT for read-only DB on {} - ignored", s3_key);
Err("Received update for read-only DB".into())
}
}
/// Function called for each incoming delete request to this storage.
async fn delete(
&mut self,
key: Option<OwnedKeyExpr>,
_timestamp: Timestamp,
) -> ZResult<StorageInsertionResult> {
let key = key.map_or_else(|| OwnedKeyExpr::from_str(NONE_KEY), Ok)?;
tracing::debug!("Delete called on client {}. Key: '{}'", self.client, key);
let s3_key = S3Key::from_key_expr(self.config.path_prefix.as_ref(), key)?;
if !self.config.is_read_only {
let key: String = s3_key.into();
let client = self.client.clone();
await_task!(client.delete_object(key).await,)
.map_err(|e| zerror!("Delete operation failed: {e}"))?;
Ok(StorageInsertionResult::Deleted)
} else {
tracing::warn!("Received DELETE for read-only DB on {} - ignored", s3_key);
Err("Received update for read-only DB".into())
}
}
async fn get_all_entries(&self) -> ZResult<Vec<(Option<OwnedKeyExpr>, Timestamp)>> {
let client = self.client.clone();
// 获取数据桶里边所有对象
let objects = await_task!(client.list_objects_in_bucket().await,)
.map_err(|e| zerror!("Get operation failed: {e}"))?;
let futures = objects.into_iter().filter_map(|object| {
let object_key = match object.key() {
Some(key) if key == NONE_KEY => return None,
Some(key) => key.to_string(),
None => {
tracing::error!("Could not get key for object {:?}", object);
return None;
}
};
match S3Key::from_key(self.config.path_prefix.as_ref(), object_key.to_owned()) {
Ok(s3_key) => {
if !s3_key.key_expr.intersects(&self.config.key_expr) {
return None;
}
}
Err(err) => {
tracing::error!("Error filtering storage entries: ${err}.");
return None;
}
};
let client = self.client.clone();
let fut = async move {
// 通过head 方法获取metadata 信息
let result = client.get_head_object(&object_key).await;
match result {
Ok(value) => {
let metadata = value.metadata.ok_or_else(|| {
zerror!("Unable to retrieve metadata for key '{}'.", object_key)
})?;
let timestamp = metadata.get(TIMESTAMP_METADATA_KEY).ok_or_else(|| {
zerror!("Unable to retrieve timestamp for key '{}'.", object_key)
})?;
let key_expr = OwnedKeyExpr::from_str(object_key.trim_start_matches('/'))
.map_err(|err| {
zerror!(
"Unable to generate key expression for key '{}': {}",
&object_key,
&err
)
})?;
Ok((
Some(key_expr),
Timestamp::from_str(timestamp.as_str()).map_err(|e| {
zerror!(
"Unable to obtain timestamp for key: {}. {:?}",
object_key,
e
)
})?,
))
}
Err(err) => Err(zerror!(
"Unable to get '{}' object from storage: {}",
object_key,
err
)),
}
};
Some(spawn_runtime(fut))
});
// 遍历处理key 时间信息
let futures_results = join_all(futures.collect::<FuturesUnordered<_>>()).await;
let entries: Vec<(Option<OwnedKeyExpr>, Timestamp)> = futures_results
.into_iter()
.flatten()
.filter_map(|x| match x {
Ok(value) => Some(value),
Err(err) => {
tracing::error!("{}", err);
None
}
})
.collect();
Ok(entries)
}
}
说明
zneoh 存储 插件的开发难度并不大，当前主流是基于rust 开发，实际上从内部机制来说是可以不限制语言的，详细s3 部分（比如配置以及s3client 包装可以查看源码）
