Quartz 4

Kafka Exactly-once semantics

3 min read

Exactly-once is a really hard problem

What is exactly-once semantics? Messaging semantics explained

  • Systems can always fail independently of one another.
  • Depending on the action the producer takes to handle such a failure, you can get different semantics:
    • At-least-once semantics If the broker had failed right before it sent the ack but after the message was successfully written to the Kafka topic, this retry leads to the message being written twice and hence delivered more than once to the end consumer.
    • At-most-once semantics if the producer does not retry when an ack times out or returns an error, then the message might end up not being written to the Kafka topic, and hence not delivered to the consumer.
    • Exactly-once semantics Exactly-once semantics is the most desirable guarantee, but also a poorly understood one. If after consuming a message successfully you rewind your Kafka consumer to a previous offset, you will receive all the messages from that offset to the latest one, all over again. This shows why the messaging system and the client application must cooperate to make exactly-once semantics happen.

Exactly-once semantics in Apache Kafka

Prior to 0.11.x, Apache Kafka supported at-least-once delivery semantics and in-order delivery per partition.

Idempotence: Exactly-once in order semantics per partition

How does this feature work? Under the covers, it works in a way similar to TCP: each batch of messages sent to Kafka will contain a sequence number that the broker will use to dedupe any duplicate send. https://developer.confluent.io/tutorials/message-ordering/kafka.html

Transactions: Atomic writes across multiple partitions

Kafka now supports atomic writes across multiple partitions through the new transactions API.This allows a producer to send a batch of messages to multiple partitions such that either all messages in the batch are eventually visible to any consumer or none are ever visible to consumers.

producer.initTransactions();
try {
  producer.beginTransaction();
  producer.send(record1);
  producer.send(record2);
  producer.commitTransaction();
} catch(ProducerFencedException e) {
  producer.close();
} catch(KafkaException e) {
  producer.abortTransaction();
}```
 
It is worth noting that a Kafka topic partition might have some messages that are part of a transaction while others are not.
 
So on the consumer side, you have two options for reading transactional messages, expressed through the isolation.level consumer config:
1. `read_committed`: In addition to reading messages that are not part of a transaction, you can also read ones that are, after the transaction is committed.
2. `read_uncommitted`: Read all messages in offset order without waiting for transactions to be committed. This option is similar to the current semantics of a Kafka consumer.
#### **The real deal: Exactly-once stream processing in Apache Kafka**
Building on idempotency and atomicity, exactly-once stream processing is now possible through the Streams API in Apache Kafka.
 
> `processing.guarantee=exactly_once`

Graph View

Backlinks