Kafka Message Delivery Guarantees: At-Least-Once, At-Most-Once, Exactly-Once

Kafka supports three delivery semantics, but the guarantee depends on how both producer and consumer are configured. On the producer side: acks=0 means fire-and-forget — at-most-once because a network error silently drops the message. acks=1 waits for the leader to ack; acks=all waits for all in-sync replicas to ack. With acks=all and retries > 0, the default behavior is at-least-once because a network timeout can cause the producer to retry and the broker to write the same message twice. The fix is enable.idempotence=true — Kafka assigns each producer a unique producerID and a monotonically increasing sequence number per partition. The broker rejects duplicate (producerID, seq) pairs, so producer retries become idempotent automatically. With idempotence on, you get exactly-once per partition from the producer. On the consumer side, the knob is offset commit timing. Commit after processing → crash between processing and commit means reprocessing on restart → at-least-once. Commit before processing → crash after commit means lost message → at-most-once. End-to-end exactly-once across a consumer → producer pipeline needs Kafka Transactions, introduced in 0.11. The producer calls beginTransaction, writes output messages, sends commit offsets for input topics, then commitTransaction — all atomic. Downstream consumers set isolation.level=read_committed so they only see messages from committed transactions, never from aborted ones. Kafka Streams uses this under the hood for exactly-once processing. In practice, very few systems need true exactly-once; most pipelines are at-least-once with an idempotent sink — dedupe by message key in the database or downstream service. That pattern is simpler and handles more failure modes than trying to get true transactional exactly-once right.