consumer-lag-triage

name	consumer-lag-triage
description	Structured triage procedure for diagnosing and resolving Kafka consumer group lag — distinguishes slow consumer, under-partitioned topic, rebalance storm, and broker-side causes.

Consumer Lag Triage

When to Use This

A consumer group's lag is growing — either a sudden spike or a steady drift. This skill produces a root cause within 15 minutes and a remediation plan before lag becomes a data-loss or SLA risk.

Step 1 — Confirm the Lag Is Real

# Snapshot lag per partition
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group <group-id>

Check columns: LOG-END-OFFSET, CURRENT-OFFSET, LAG. A lag that is constant (not growing) is a healthy steady state. Proceed only if lag is increasing over successive snapshots.

Step 2 — Narrow the Cause

Observable	Most likely cause
Lag on ALL partitions, all consumers	Slow consumer processing; broker throughput
Lag on ONE partition, others fine	Hot partition; single slow consumer instance
Lag spikes then recovers repeatedly	Rebalance storm (frequent group rejoins)
Lag growing with zero consumer activity	Consumer is down / crashed
Lag growing despite high consumer throughput	Under-partitioned topic; need more partitions

Step 3 — Diagnose by Cause

Slow consumer processing:

• Profile the consumer's poll() → processing → commit() loop. Processing time must be < max.poll.interval.ms (default 5 min); exceeding it triggers a rebalance.
• Check fetch.max.bytes and max.partition.fetch.bytes — if consumers pull large batches but process slowly, reduce batch size or offload processing asynchronously.

Rebalance storm:

• Check kafka.consumer:type=consumer-coordinator-metrics,attribute=rebalance-rate metric.
• Common causes: GC pauses exceeding session.timeout.ms; deployment rolling restarts without graceful leave; too many partitions per consumer (> ~200 triggers slow rebalance).
• Fix: tune session.timeout.ms / heartbeat.interval.ms, enable static membership (group.instance.id) to survive restarts without rebalance.

Hot partition:

• Inspect key distribution with a short consumer that counts by key. Skew means the partition key doesn't distribute work.
• Options: re-key, add a synthetic random suffix for pure-parallel work, or use a custom partitioner.

Broker-side:

• Check broker CPU, disk I/O (kafka.server:type=BrokerTopicMetrics), and network saturation.
• Unclean leader elections or under-replicated partitions (UnderReplicatedPartitions > 0) introduce read stalls.

Step 4 — Remediation Decision

Is the consumer keeping up when healthy (lag stable)?
  Yes → scale consumer instances (add pods/threads up to partition count)
  No  → is processing CPU/IO bound?
    Yes → offload or parallelize processing; check batch size
    No  → check rebalance; tune heartbeat/session timeouts
Is partition count < desired parallelism?
  Yes → increase partitions (irreversible; plan ordering impact)
  No  → review consumer group assignment strategy (cooperative-sticky preferred)

Step 5 — Reset Offsets (Last Resort)

# Dry run first
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group <group-id> --reset-offsets --to-latest --topic <topic> --dry-run
# Execute
kafka-consumer-groups.sh ... --execute

Resetting to --to-latest drops all un-consumed messages. Only do this when you've confirmed the data is recoverable from another source or lag recovery is impossible.

Pitfalls

• Increasing partition count on a keyed topic changes which partition a key maps to — messages for the same key can appear in a different partition after the increase, breaking per-key ordering.
• Adding consumer instances beyond the partition count — extra instances sit idle and contribute to rebalance overhead.
• Setting max.poll.interval.ms very high to mask slow processing — this hides real issues and delays rebalance detection when a consumer truly dies.
• Committing offsets before processing completes (fire-and-forget) — lag looks fine while data is actually lost on crash.

See Also

• ../../agents/stream-processing-engineer.md — backpressure and stateful processing
• ../../agents/kafka-pipeline-engineer.md — partition design and consumer group ownership