C#：kafka

在本篇博客中，我们将介绍如何在 C# 项目中使用 Kafka。Kafka 是一个分布式流处理平台，用于构建实时数据流管道和应用程序。我们将使用 Confluent.Kafka 库来实现 C# 调用 Kafka。
1.准备工作
首先，确保已经安装了 .NET Core SDK。接下来，我们需要在项目中安装 Confluent.Kafka 库。在项目文件夹中打开终端，运行以下命令： 

dotnet add package Confluent.Kafka

2.生产者示例
下面是一个简单的生产者示例，用于将消息发送到 Kafka 主题。

using System;
using System.Threading.Tasks;
using Confluent.Kafka;

class Program
{
    public static async Task Main(string[] args)
    {
        var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

        using (var producer = new ProducerBuilder<Null, string>(config).Build())
        {
            try
            {
                var dr = await producer.ProduceAsync("test-topic", new Message<Null, string> { Value = "Hello, Kafka!" });
                Console.WriteLine($"Delivered '{dr.Value}' to '{dr.TopicPartitionOffset}'");
            }
            catch (ProduceException<Null, string> e)
            {
                Console.WriteLine($"Delivery failed: {e.Error.Reason}");
            }
        }
    }
}

3.消费者示例
下面是一个简单的消费者示例，用于从 Kafka 主题中读取消息。 

using System;
using System.Threading;
using Confluent.Kafka;

class Program
{
    public static void Main(string[] args)
    {
        var conf = new ConsumerConfig
        {
            GroupId = "test-consumer-group",
            BootstrapServers = "localhost:9092",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using (var consumer = new ConsumerBuilder<Ignore, string>(conf).Build())
        {
            consumer.Subscribe("test-topic");

            CancellationTokenSource cts = new CancellationTokenSource();
            Console.CancelKeyPress += (_, e) => {
                e.Cancel = true;
                cts.Cancel();
            };

            try
            {
                while (true)
                {
                    try
                    {
                        var cr = consumer.Consume(cts.Token);
                        Console.WriteLine($"Consumed message '{cr.Value}' at: '{cr.TopicPartitionOffset}'.");
                    }
                    catch (ConsumeException e)
                    {
                        Console.WriteLine($"Error occured: {e.Error.Reason}");
                    }
                }
            }
            catch (OperationCanceledException)
            {
                consumer.Close();
            }
        }
    }
}

 

1.手动设置TopicPartition=>offset

//手动设置TopicPartition=>offset
foreach (TopicPartition partition in consumer.Assignment)
{
    if (partition.Partition.Value == 13)
    {
        TopicPartitionOffset offset = new TopicPartitionOffset(partition, 27060);
        consumer.Seek(offset);
    }
}

2.手动指定kafka的Partition

List<TopicPartition> list = new List<TopicPartition>();
for (int i = 0; i < 30; i++)
{
    TopicPartition p = new TopicPartition("test.topic", new Partition(i));
    list.Add(p);
}
consumer.Assign(list);