The Day Our Kafka Consumer Lag Hit 2 Million Messages
A config change nobody reviewed turned a calm Thursday into an eleven hour incident, and here's the full postmortem.
At 10.42 on a Thursday morning, our on-call channel lit up with an alert I'd never seen fire before. Consumer lag on the order events topic was 180,000 messages and climbing.
By lunch it was 2.1 million.
Orders were still being placed. Customers were getting charged. But confirmation emails, warehouse picks, and loyalty points were all stuck behind a wall of unprocessed events. Here's what happened, what we got wrong, and what we changed.
The Setup
Our order events topic had 24 partitions. The consumer was a Spring Boot service using Spring Kafka, running 6 pods with 4 consumer threads each. One thread per partition.
On a normal day, we processed around 1,500 messages per second at peak, and lag stayed under 500. Nobody looked at this service much. It just worked.
That's usually when things go wrong.
The First Hour. Chasing the Wrong Thing
My first guess was a traffic spike. It wasn't. Producer rates looked normal.
My second guess was a slow downstream. Our consumer called the warehouse API and a loyalty service for each event. I checked their dashboards. Both were healthy with normal latency.
Then I noticed the weird part. Consumer throughput wasn't low. It was near zero, and it came in bursts. A few thousand messages, then nothing for about 5 minutes, then a few thousand more.
That pattern smelled like rebalancing.
When consumer throughput comes in bursts with long gaps, stop looking at your code and start looking at your group coordinator logs.
The Rebalance Storm
The consumer logs confirmed it. Every few minutes, one consumer was getting kicked out of the group with this message about exceeding max.poll.interval.ms. That triggered a rebalance. During a rebalance, every consumer in the group stops processing.
When the group stabilized, a different consumer would get kicked out. Then another rebalance. We were spending most of our time rebalancing and almost none processing.
But why were consumers exceeding the poll interval? Downstream latency was fine.
The Root Cause
Two days earlier, someone had raised max.poll.records from 100 to 1,000. The pull request said "improve throughput." It was a one line config change, and it got approved in about four minutes.
Our processing time per message was around 280ms, mostly from the two downstream calls. With 100 records per poll, that's 28 seconds per batch. Well under the 5 minute default poll interval.
With 1,000 records, it's 280 seconds. Just under 5 minutes on a good batch.
On a bad batch, where a few downstream calls took 600ms instead of 250ms, it went over. The consumer got kicked. The rebalance started. Its partitions moved to another consumer, which now had more work and was also close to the limit. That consumer got kicked next.
Thursday morning just had slightly more slow calls than usual. That was enough to tip it over.
How We Stopped the Bleeding
First, we reverted max.poll.records to 100. That stopped the rebalance storm within about 10 minutes.
Get Naveen Metta’s stories in your inbox
Join Medium for free to get updates from this writer.
But we still had 2 million messages queued, and at 1,500 per second that's over 20 minutes in theory. In practice it was way slower because the backlog caused heavier downstream load.
So we did a few things in order.
- Scaled from 6 to 24 consumer threads total so every partition had its own thread
- Temporarily disabled the loyalty points call and queued those events to a separate topic for later replay
- Asked the warehouse team to scale their API for a few hours
We couldn't go past 24 consumers, because partitions cap your parallelism. More pods than partitions just means idle pods. That's a lesson I'd love to forget learning.
Lag was back under 1,000 at 9.50 that night. About 11 hours total.
The Code Change That Mattered Most
The real problem was that our processing was slow and serial. Each message made two blocking HTTP calls, one after the other.
We changed the consumer to process records from a single poll in parallel, while keeping order per key. Order events for the same order ID must stay in sequence, so we grouped by key and ran groups concurrently.
@KafkaListener(topics = "order-events", batch = "true")
public void onBatch(List<ConsumerRecord<String, OrderEvent>> records) {
Map<String, List<ConsumerRecord<String, OrderEvent>>> byKey =
records.stream().collect(groupingBy(ConsumerRecord::key));
List<CompletableFuture<Void>> work = byKey.values().stream()
.map(group -> CompletableFuture.runAsync(
() -> group.forEach(this::process), workerPool))
.toList();
CompletableFuture.allOf(work.toArray(new CompletableFuture[0])).join();
}Batch processing time dropped from 28 seconds to about 4. We could have used Confluent's parallel consumer library too, and we probably will eventually. This was the smallest change we could ship safely that week.
What We Changed Afterward
We added alerts on consumer lag growth rate, not just absolute lag. A lag of 50,000 that's shrinking is fine. A lag of 5,000 that's doubling every minute is an incident.
We added alerts on rebalance frequency. More than 2 rebalances in 10 minutes now pages someone.
We started tracking processing time per batch as a Micrometer timer and alerting at 50% of max.poll.interval.ms. That gives us warning long before a consumer gets kicked.
We also enabled cooperative sticky assignment. With the old eager protocol, every rebalance stopped all consumers. With cooperative rebalancing, only the partitions that actually move get paused. It doesn't fix the root cause, but it makes the blast radius much smaller.
And we made a team rule. Any change to Kafka consumer config needs a short note in the PR explaining the math. How long does a batch take? How close is that to the poll interval?
What I'd Tell Past Me
The config change wasn't the real failure. The real failure was that nobody understood how max.poll.records, processing time, and max.poll.interval.ms relate to each other. Including me, honestly, until that Thursday.
Kafka is forgiving right up until it isn't. Then it fails in a way that looks totally unrelated to the change that caused it.
Key takeaways
- Batch processing time must stay well under the poll interval, so do the math
- Bursty consumer throughput with long gaps usually means rebalancing
- Partition count caps your consumer parallelism
- Alert on lag growth rate and rebalance frequency, not just lag
- Use cooperative sticky assignment to shrink rebalance pain
- Treat consumer config changes like code changes
What's the worst Kafka incident you've lived through? I'd honestly like to compare notes in the comments.
