NATS: 对象存储
https://natsbyexample.com/examples/os/intro/dotnet2
NATS 中的对象存储能力是在 Stream 之上的一种抽象，它将消息的主题视为类似键值对中的键，但是其中的内容可以跨越多个数据块。使得它支持大型数据。并且通常作为可读/可写流进行加载和读取。
比较来说，Key-Value 存储限制保存的数据量为 1M, 而对象存储没有这个限制。
注意：对象存储不是分布式存储系统。所有存储的文件需要适应目标文件系统。
它支持客户端应用程序创建并存储一组文件的存储桶（对应于流）。文件以块的形式存储和传输，允许通过 NATS 基础设施安全地传输任意大小的文件。
1. 创建对象存储管理器
使用 JSContext 来创建对象存储的管理器 NatsObjContext
var opts = new NatsOpts
{
Url = url,
LoggerFactory = loggerFactory,
Name = "NATS-by-Example",
};
await using var nats = new NatsConnection(opts);
var js = new NatsJSContext(nats);
var objMgr = new NatsObjContext(js);
2. 创建命名的对象存储
借助对象管理器来创建一个命名的对象存储。每个对象存储可以看做一个字典。
创建时，可以指定一个存储的配置
/// <summary>
/// Object store configuration.
/// </summary>
/// <param name="Bucket">Name of the bucket.</param>
public record NatsObjConfig(string Bucket)
{
/// <summary>
/// Bucket description.
/// </summary>
public string? Description { get; init; }
/// <summary>
/// Maximum age of the object.
/// </summary>
public TimeSpan? MaxAge { get; init; }
/// <summary>
/// How big the store may be, when the combined stream size exceeds this old keys are removed.
/// </summary>
public long? MaxBytes { get; init; }
/// <summary>
/// Type of backing storage to use.
/// </summary>
public NatsObjStorageType Storage { get; init; }
/// <summary>
/// How many replicas to keep for each key.
/// </summary>
public int NumberOfReplicas { get; init; } = 1;
/// <summary>
/// Placement requirements for a object store stream.
/// </summary>
public Placement? Placement { get; init; }
/// <summary>
/// Additional metadata for the bucket.
/// </summary>
public Dictionary<string, string>? Metadata { get; init; }
/// <summary>
/// Use compressed storage.
/// </summary>
public bool Compression { get; init; }
}
创建对象存储
var store = await objMgr.CreateObjectStore("configs");
INatsObjStore
ublic interface INatsObjStore
{
//
// Summary:
// 提供访问底层的 JetStream context
INatsJSContext JetStreamContext { get; }
//
// Summary:
// Object store 的 bucket name.
string Bucket { get; }
//
// Summary:
// 通过 Key 来提取其中保存的对象.
//
// Parameters:
// key:
// 对象的 key.
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// 以字节数组的形式返回对象的值.
ValueTask<byte[]> GetBytesAsync(string key, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 通过 Key 提取对象.
//
// Parameters:
// key:
// 对象的 key.
//
// stream:
// 用来将对象的值写入的流.
//
// leaveOpen:
// true 表示在异步方法返回的时候，不会关闭底层的流
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// 直接返回的是对象关联的 metadata 数据对象.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// Metadata didn't match the value retrieved e.g. the SHA digest.
ValueTask<ObjectMetadata> GetAsync(string key, Stream stream, bool leaveOpen = false, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 压入对象，提供对象的 Key
//
// Parameters:
// key:
// 对象的 key.
//
// value:
// 字节数组形式的对象值.
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Object metadata.
ValueTask<ObjectMetadata> PutAsync(string key, byte[] value, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 压入对象，提供对象的 Key.
//
// Parameters:
// key:
// Object key.
//
// stream:
// 对象的值以 Stream 的形式提供.
//
// leaveOpen:
// true ，当一部方法返回的时候，不会关闭底层的流
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// 返回对象关联的 Object metadata.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// There was an error calculating SHA digest.
//
// T:NATS.Client.JetStream.NatsJSApiException:
// Server responded with an error.
ValueTask<ObjectMetadata> PutAsync(string key, Stream stream, bool leaveOpen = false, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 压入对象，提供 Key.
//
// Parameters:
// meta:
// 同时提供对象的 metadata.
//
// stream:
// 以流方式提供对象值.
//
// leaveOpen:
// true to not close the underlying stream when async method returns; otherwise,
// false
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Object metadata.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// There was an error calculating SHA digest.
//
// T:NATS.Client.JetStream.NatsJSApiException:
// Server responded with an error.
ValueTask<ObjectMetadata> PutAsync(ObjectMetadata meta, Stream stream, bool leaveOpen = false, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 更新对象的 metadata
//
// Parameters:
// key:
// 对象的 key
//
// meta:
// 对象的 metadata
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// 返回对象的 metadata
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// There is already an object with the same name
ValueTask<ObjectMetadata> UpdateMetaAsync(string key, ObjectMetadata meta, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 添加到另外对象的链接
//
// Parameters:
// link:
// 链接的名称 name
//
// target:
// 目标对象的 name
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Metadata of the new link object
ValueTask<ObjectMetadata> AddLinkAsync(string link, string target, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// Add a link to another object
//
// Parameters:
// link:
// Link name
//
// target:
// Target object's metadata
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Metadata of the new link object
ValueTask<ObjectMetadata> AddLinkAsync(string link, ObjectMetadata target, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// Add a link to another object store
//
// Parameters:
// link:
// Object's name to be linked
//
// target:
// Target object store
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Metadata of the new link object
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// Object with the same name already exists
ValueTask<ObjectMetadata> AddBucketLinkAsync(string link, INatsObjStore target, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 密封该存储 the object store. No further modifications will be allowed.
//
// Parameters:
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// Update operation failed
ValueTask SealAsync(CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// Get object metadata by key.
//
// Parameters:
// key:
// Object key.
//
// showDeleted:
// Also retrieve deleted objects.
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Object metadata.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// Object was not found.
ValueTask<ObjectMetadata> GetInfoAsync(string key, bool showDeleted = false, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 列出存储中的对象.
//
// Parameters:
// opts:
// List options
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// An async enumerable object metadata to be used in an await foreach
IAsyncEnumerable<ObjectMetadata> ListAsync(NatsObjListOpts? opts = null, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// Retrieves run-time status about the backing store of the bucket.
// 提取此 Bucket 后台存储的状态
//
// Parameters:
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// Object store status
ValueTask<NatsObjStatus> GetStatusAsync(CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// Watch for changes in the underlying store and receive meta information updates.
// 监控底层的存储，接收元信息的更新
//
// Parameters:
// opts:
// Watch options
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Returns:
// An async enumerable object metadata to be used in an await foreach
// 返回一个异步的对于对象元数据的迭代器，用来进行 await foreach 操作
IAsyncEnumerable<ObjectMetadata> WatchAsync(NatsObjWatchOpts? opts = null, CancellationToken cancellationToken = default(CancellationToken));
//
// Summary:
// 通过 Key 删除存储的对象
//
// Parameters:
// key:
// Object key.
//
// cancellationToken:
// A System.Threading.CancellationToken used to cancel the API call.
//
// Exceptions:
// T:NATS.Client.ObjectStore.NatsObjException:
// Object metadata was invalid or chunks can't be purged.
ValueTask DeleteAsync(string key, CancellationToken cancellationToken = default(CancellationToken));
}
3. 检查对象存储的状态
使用对象存储的 GetStatusAsync() 可以检查它的状态
var status = await store.GetStatusAsync();
logger.LogInformation("The object store has {Size} bytes", status.Info.State.Bytes);
Output
info: NATS-by-Example[0]
The object store has 0 bytes
4. 存储数据和元数据
使用 PutAsync() 将数据存储到对象存储中。除了数据本身之外，还可以附加描述数据的元数据。
如果重复使用相同的 Key 存储数据，新的数据将会覆盖旧的数据。
每块数据关联一个元数据的描述，可以通过它对数据进行说明。
const int bytes = 10_000_000;
var data = new byte[bytes];
var info = await store.PutAsync(key: "a", data);
logger.LogInformation("Added entry {Name} ({Size} bytes)- '{Description}'", info.Name, info.Size, info.Description);
await store.UpdateMetaAsync("a", new ObjectMetadata { Name = "a", Description = "still large data" });
Output
info: NATS-by-Example[0]
Added entry a (10000000 bytes)- '(null)'
ObjectMetadata 有一个 Headers 的属性，支持字符串形式的键值对说明。
public Dictionary<string, string[]>? Headers { get; set; }
ObjectMetadata 的定义
public record ObjectMetadata
{
/// <summary>
/// Object name
/// </summary>
[JsonPropertyName("name")]
#if NET6_0
public string Name { get; set; } = default!;
#else
#pragma warning disable SA1206
public required string Name { get; set; }
#pragma warning restore SA1206
#endif
/// <summary>
/// Object description
/// </summary>
[JsonPropertyName("description")]
public string? Description { get; set; }
/// <summary>
/// Bucket name
/// </summary>
[JsonPropertyName("bucket")]
public string? Bucket { get; set; }
/// <summary>
/// Object NUID
/// </summary>
[JsonPropertyName("nuid")]
public string? Nuid { get; set; }
/// <summary>
/// Max chunk size
/// </summary>
[JsonPropertyName("size")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public int Size { get; set; }
/// <summary>
/// Modified timestamp
/// </summary>
[JsonPropertyName("mtime")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public DateTimeOffset MTime { get; set; }
/// <summary>
/// Number of chunks
/// </summary>
[JsonPropertyName("chunks")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public int Chunks { get; set; }
/// <summary>
/// Object digest
/// </summary>
[JsonPropertyName("digest")]
public string? Digest { get; set; }
/// <summary>
/// Object metadata
/// </summary>
[JsonPropertyName("metadata")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public Dictionary<string, string>? Metadata { get; set; }
/// <summary>
/// Object metadata
/// </summary>
[JsonPropertyName("headers")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public Dictionary<string, string[]>? Headers { get; set; }
/// <summary>
/// Object deleted
/// </summary>
[JsonPropertyName("deleted")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public bool Deleted { get; set; }
/// <summary>
/// Object options
/// </summary>
[JsonPropertyName("options")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public MetaDataOptions? Options { get; set; }
}
5. 提取存储的数据
使用键值来提取存储的数据。
var data1 = await store.GetBytesAsync("a");
logger.LogInformation("Data has {Size} bytes", data1.Length);
Output
info: NATS-by-Example[0]
Data has 10000000 bytes
6. 删除对象存储
借助键值可以删除存储的数据。
await store.DeleteAsync("a");
7. 以字节流的方式存储和读取数据
可以使用字节流对象，而不是字节数组来存储和提取数据。
const int bytes = 10_000_000;
var data = new byte[bytes];
// 使用字节流提供数据
info = await store.PutAsync(new ObjectMetadata { Name = "b", Description = "set with a stream" }, new MemoryStream(data));
logger.LogInformation("Added entry {Name} ({Size} bytes)- '{Description}'", info.Name, info.Size, info.Description);
// 使用字节流提取数据，数据被填充到流中
var ms = new MemoryStream();
info = await store.GetAsync("b", ms);
logger.LogInformation("Got entry {Name} ({Size} bytes)- '{Description}'", info.Name, info.Size, info.Description);
await obj.DeleteObjectStore("configs", CancellationToken.None);
Output
info: NATS-by-Example[0]
Added entry b (10000000 bytes)- 'set with a stream'
info: NATS-by-Example[0]
Got entry b (10000000 bytes)- 'set with a stream'
8.管理对象存储
列出存储的数据
var count = 0;
await foreach (var entry in store.ListAsync())
{
logger.LogInformation("Entry {Name} ({Size} bytes)- '{Description}'", info.Name, info.Size, info.Description);
count++;
}
logger.LogInformation("The object store contains {Count} entries", count);
Output
info: NATS-by-Example[0]
Entry a (10000000 bytes)- '(null)'
info: NATS-by-Example[0]
The object store contains 1 entries
9. 监控对象存储中的变更
var watcher = Task.Run(async () =>
{
await foreach (var m in store.WatchAsync(new NatsObjWatchOpts{IncludeHistory = false}))
{
logger.LogInformation(">>>>>>>> Watch: {Bucket} changed '{Name}' {Op}", m.Bucket, m.Name, m.Deleted ? "was deleted" : "was updated");
}
});
Output
info: NATS-by-Example[0]
>>>>>>>> Watch: configs changed 'a' was updated
通过 NatsObjWatchOpts 可以配置监听的内容
- IgnoreDeletes 是否忽略删除操作
- 是否包含历史操作
- 是否仅仅监听更新操作
- 是否仅仅监听初始化操作，并忽略后继的更新
还可以提供一个回调 Func<> 来指定当异步枚举完成的时候，通过返回 true 来结束异步枚举，返回 false 来继续枚举。
public record NatsObjWatchOpts
{
public bool IgnoreDeletes { get; init; }
public bool IncludeHistory { get; init; }
public bool UpdatesOnly { get; init; }
/// <summary>
/// Only return the initial set of objects and don't watch for further updates.
/// </summary>
public bool InitialSetOnly { get; init; }
/// <summary>
/// Async function called when the enumerator reaches the end of data. Return True to break the async enumeration, False to allow the enumeration to continue.
/// </summary>
public Func<CancellationToken, ValueTask<bool>>? OnNoData { get; init; }
}
10. 使用 NATS CLI 管理对象存储
NATS CLI 提供了 object 命令管理对象存储
创建对象存储
nats object add myobjbucket
保存文件
nats object put myobjbucket ~/Movies/NATS-logo.mov
列出保存的对象
nats object ls myobjbucket
Deep
{
"type":"io.nats.jetstream.api.v1.stream_msg_get_response",
"message":
{
"subject":"$O.Report.M.ODQzZjllZTEtNDM1MC00Y2JmLWE5M2UtZmZiODE1OWQ2ZWI4",
"seq":11060,
"hdrs":"TkFUUy8xLjANCk5hdHMtUm9sbHVwOiBzdWINCg0K",
"data":"eyJuYW1lIjoiODQzZjllZTEtNDM1MC00Y2JmLWE5M2UtZmZiODE1OWQ2ZWI4IiwiZGVzY3JpcHRpb24iOm51bGwsImJ1Y2tldCI6IlJlcG9ydCIsIm51aWQiOiJNMWhyZTcwUmFXRURQWld4SVRnVGVXIiwibXRpbWUiOiIyMDI1LTAyLTE5VDA4OjI3OjI0LjkwOTczODgrMDA6MDAiLCJkaWdlc3QiOiJTSEEtMjU2PTQ3REVRcGo4SEJTYS1fVEltVy01SkNldVFlUmttNU5NcEpXWkczaFN1RlU9IiwiaGVhZGVycyI6eyJzdGF0dXMiOlsiY3JlYXRlZCJdfSwib3B0aW9ucyI6eyJtYXhfY2h1bmtfc2l6ZSI6MTMxMDcyfX0=",
"time":"2025-02-19T08:27:24.9437205Z"
}
}
data was Base64 Encoded Message
{
"name":"843f9ee1-4350-4cbf-a93e-ffb8159d6eb8",
"description":null,
"bucket":"Report",
"nuid":"M1hre70RaWEDPZWxITgTeW",
"mtime":"2025-02-19T08:27:24.9097388+00:00",
"digest":"SHA-256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU=",
"headers":
{
"status":["created"]
},
"options":
{
"max_chunk_size":131072
}
}
Bucket Info
> ./nats object info Report
Information for Object Store Bucket Report created 2025-02-13T15:13:46+08:00
Configuration:
Bucket Name: Report
Description: Report
Replicas: 1
TTL: 10m0s
Sealed: false
Size: 481 B
Maximum Bucket Size: unlimited
Backing Store Kind: JetStream
JetStream Stream: OBJ_Report
Cluster Information:
Name:
Leader: NC5HLMZWGOTF2C424QLY6T6GWLZB3NI4IVOHBGUIE7QMVGBB3CMHOBLY
Object Info
> ./nats object info Report 843f9ee1-4350-4cbf-a93e-ffb8159d6eb8
Object information for Report > 843f9ee1-4350-4cbf-a93e-ffb8159d6eb8
Size: 0 B
Modification Time: 2025-02-19 16:27:24
Chunks: 0
Digest: SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Headers: status: created
参考资料
合集:
NATS
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 单线程的Redis速度为什么快？