C#:kafka
在本篇博客中,我们将介绍如何在 C# 项目中使用 Kafka。Kafka 是一个分布式流处理平台,用于构建实时数据流管道和应用程序。我们将使用 Confluent.Kafka 库来实现 C# 调用 Kafka。
1.准备工作
首先,确保已经安装了 .NET Core SDK。接下来,我们需要在项目中安装 Confluent.Kafka 库。在项目文件夹中打开终端,运行以下命令:
dotnet add package Confluent.Kafka
2.生产者示例
下面是一个简单的生产者示例,用于将消息发送到 Kafka 主题。
using System; using System.Threading.Tasks; using Confluent.Kafka; class Program { public static async Task Main(string[] args) { var config = new ProducerConfig { BootstrapServers = "localhost:9092" }; using (var producer = new ProducerBuilder<Null, string>(config).Build()) { try { var dr = await producer.ProduceAsync("test-topic", new Message<Null, string> { Value = "Hello, Kafka!" }); Console.WriteLine($"Delivered '{dr.Value}' to '{dr.TopicPartitionOffset}'"); } catch (ProduceException<Null, string> e) { Console.WriteLine($"Delivery failed: {e.Error.Reason}"); } } } }
3.消费者示例
下面是一个简单的消费者示例,用于从 Kafka 主题中读取消息。
using System; using System.Threading; using Confluent.Kafka; class Program { public static void Main(string[] args) { var conf = new ConsumerConfig { GroupId = "test-consumer-group", BootstrapServers = "localhost:9092", AutoOffsetReset = AutoOffsetReset.Earliest }; using (var consumer = new ConsumerBuilder<Ignore, string>(conf).Build()) { consumer.Subscribe("test-topic"); CancellationTokenSource cts = new CancellationTokenSource(); Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); }; try { while (true) { try { var cr = consumer.Consume(cts.Token); Console.WriteLine($"Consumed message '{cr.Value}' at: '{cr.TopicPartitionOffset}'."); } catch (ConsumeException e) { Console.WriteLine($"Error occured: {e.Error.Reason}"); } } } catch (OperationCanceledException) { consumer.Close(); } } } }
1.手动设置TopicPartition=>offset
//手动设置TopicPartition=>offset foreach (TopicPartition partition in consumer.Assignment) { if (partition.Partition.Value == 13) { TopicPartitionOffset offset = new TopicPartitionOffset(partition, 27060); consumer.Seek(offset); } }
2.手动指定kafka的Partition
List<TopicPartition> list = new List<TopicPartition>(); for (int i = 0; i < 30; i++) { TopicPartition p = new TopicPartition("test.topic", new Partition(i)); list.Add(p); } consumer.Assign(list);