Kafka consumer groups and dead letter queues

A consumer group is how Kafka spreads a topic across workers. A dead letter queue is how those workers avoid getting stuck on a message they cannot process. The two are usually configured together and usually configured wrong.

Originally written on 7 October 2019. Migrated from a WordPress blog and reformatted.

Consumer groups

A topic is divided into partitions. A consumer group is a set of consumers that share a group.id; Kafka assigns each partition to exactly one consumer in the group. Two consequences follow immediately.

Parallelism is bounded by partition count. A topic with 6 partitions and a group of 8 consumers leaves 2 consumers idle. Adding consumers beyond the partition count does nothing.

Ordering holds within a partition only. Messages with the same key go to the same partition and are consumed in order. Messages with different keys may be processed in any relative order. If ordering matters for an entity, the entity's id must be the message key.

Each consumer tracks its position per partition as an offset and commits it back to Kafka. On restart, or when a partition is reassigned during a rebalance, the new consumer resumes from the last committed offset. Everything after that offset is redelivered, which is why every consumer must tolerate duplicates.

Offset commit timing

enable.auto.commit=true (the default) commits offsets on a timer, regardless of whether the messages have been processed. A consumer that polls 500 messages, auto-commits, and crashes at message 200 loses 300 messages. For anything that matters, commit manually after processing:

props.put("enable.auto.commit", "false");
...
for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
    process(r);
}
consumer.commitSync();

This gives at-least-once delivery: a crash between process and commitSync redelivers the batch. Combined with idempotent processing it is the standard configuration.

Rebalances

When a consumer joins or leaves the group, Kafka reassigns partitions. During the rebalance, no consumer in the group processes anything. Frequent rebalances, caused by consumers that exceed max.poll.interval.ms between polls because processing is slow, are the most common cause of a consumer group that appears to hang. The fix is to reduce max.poll.records, increase max.poll.interval.ms, or move slow work off the poll thread.

The poison message

A message that cannot be processed, because it is malformed, references a record that does not exist, or triggers a bug, will be retried on every redelivery. Since the offset cannot advance past it, the partition stops. Every message behind it waits indefinitely.

The remedy is a dead letter queue: a separate topic to which unprocessable messages are sent, after which the offset advances.

try {
    process(r);
} catch (NonRetryableException e) {
    deadLetterProducer.send(new ProducerRecord<>("orders.dlq", r.key(), r.value(),
        headersWith(r.headers(), "error", e.getMessage(), "original-offset", r.offset())));
}

What belongs in the dead letter queue

Only failures that will not succeed on retry. Distinguishing them from transient failures is the whole design problem:

Transient failures should not go to the DLQ on first occurrence, or the DLQ fills with messages that would have succeeded a second later. Bounded retries with backoff first, DLQ as the terminal state.

Retrying without blocking the partition

Retrying in place, by not committing the offset, blocks the partition for the duration of the backoff. The alternative is a retry topic: on failure, publish the message to orders.retry with a delay header, commit the original offset, and have a separate consumer process the retry topic after the delay. The main partition keeps moving.

Operating the dead letter queue

A DLQ that nobody reads is a slower way of losing messages. It needs a consumer that alerts on depth, a tool to inspect messages with their error headers, and a way to replay them to the original topic after the underlying fault is fixed. Without the replay path, every message in the DLQ is a manual recovery.