properties.enable.idempotence 消息顺序和消息不重复

处理：

在kafka 结果表with 参数里加'properties.enable.idempotence'='false' 来关闭客户端的幂等/事务

Kafka producer configuration reference | Confluent Documentation https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html

 

How to maintain message ordering and no message duplication using Kafka https://developer.confluent.io/tutorials/message-ordering/kafka.html

Question:

How can you maintain the order of messages and prevent message duplication in a Kafka topic partition?

Example use case:

If your application needs to maintain message ordering and prevent duplication, you can enable idempotency for your Apache Kafka producer. An idempotent producer has a unique producer ID and uses sequence IDs for each message, allowing the broker to ensure, on a per-partition basis, that it is committing ordered messages with no duplication.

Hands-on code example:

Basic Kafka

Short Answer

 

Set the ProducerConfig configuration parameters relevant to the idempotent producer:

enable.idempotence=true
acks=all

 

 Apache Kafka https://kafka.apache.org/28/documentation.html#consumerconfigs

 

enable.idempotence

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConfigException will be thrown.

Type:	boolean
Default:	false
Valid Values:
Importance:	low

 

 

 

 

 

翻译

搜索

复制