How to Triage Kafka Consumer Lag with Kafka's Native Tools

One command answers most Kafka consumer lag questions before you open a single dashboard:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app

Four columns in its output do the heavy lifting: CURRENT-OFFSET (what the group has committed), LOG-END-OFFSET (what the broker has accepted), LAG (the difference), and CONSUMER-ID (who owns the partition, or a dash if nobody does). Learn to read those four and you can triage most lag incidents with tooling that ships in every Kafka distribution.

In part one of this series we covered the six signals that matter: what lag actually measures, the three lag shapes, rebalancing storms, partition skew, under-replicated partitions, and retention pressure. This article is the hands-on triage session: every check, using only kafka-consumer-groups.sh, kafka-topics.sh, kafka-log-dirs.sh, consumer JMX metrics, and their Confluent Cloud equivalents. Budget an hour for a first pass on a cluster you've never profiled.

01Tool 1

Group table

kafka-consumer-groups.sh: per-partition lag, assignment, group state.

02Tool 2

Broker side

kafka-topics.sh and kafka-log-dirs.sh: replication health and partition sizes.

03Tool 3

JMX trends

Consumer fetch and coordinator metrics: rates, not snapshots.

04Tool 4

Tune

The timeout triangle, cooperative rebalancing and static membership.

The order this guide follows. Rule out the broker before you touch a consumer config.

Tool 1: kafka-consumer-groups.sh, the lag view

This script covers lag measurement, skew detection and rebalance forensics in one place (Kafka docs).

Step 1: Read the per-partition lag table

Start with the describe from the intro. A healthy group looks boring: small, stable lag spread evenly across partitions.

GROUP        TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID
payments-app orders  0          482190          482201          11    consumer-1-...
payments-app orders  1          479002          479020          18    consumer-1-...
payments-app orders  2          481575          481590          15    consumer-2-...

Three unhealthy patterns to spot immediately:

Lag on every partition, growing on repeat runs. The group is under-provisioned or the processing path got slower. Run the command twice, 60 seconds apart, and diff the LAG column. Direction matters more than the absolute number.
Lag concentrated on one or two partitions. Partition skew. One hot key (a big tenant, or null keys before Kafka 2.4's sticky partitioner) is funneling traffic to one partition, and no amount of consumer scaling fixes it, because a partition has exactly one owner.
A dash in CONSUMER-ID. Unassigned partitions. Either the group has fewer members than partitions and something crashed, or you're mid-rebalance.

To sweep every group on the cluster rather than one you already suspect:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --all-groups | awk '$6 > 10000'

Step 2: Check assignment balance with --members --verbose

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app --members --verbose

This prints each member, its #PARTITIONS count and the exact assignment list. What you want: partition counts within one of each other. What you often find: one member holding six partitions while its peers hold two, a leftover from eager rebalancing plus uneven subscription, and a structural reason one consumer lags while the rest idle.

Step 3: Check group state and rebalance history

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app --state

Stable is the only state you want to see in steady operation. PreparingRebalance or CompletingRebalance on repeated runs means the group is churning: members join and leave faster than assignments settle, which pauses consumption every cycle (fully, under the default eager protocol). That's your cue to jump to the rebalancing triangle below.

One honest caveat: LAG here is computed from committed offsets. A consumer that's processing fine but committing every 30 seconds will show phantom lag up to 30 seconds deep. Check auto.commit.interval.ms (or your manual commit cadence) before declaring an incident.

Tool 2: kafka-topics.sh and kafka-log-dirs.sh, the broker side

Consumer-side lag with a broker-side cause is common enough that this check belongs in every triage.

Under-replicated partitions

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --under-replicated-partitions

Empty output is the pass condition. Any rows mean followers are falling behind leaders (a broker under I/O or network pressure), and consumers reading from that broker will lag no matter how healthy the group is. Pair it with the unavailable-partitions check while you're here:

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --unavailable-partitions

Uneven partition sizes

Skew shows up on disk before it shows up in lag. Dump per-partition log sizes:

bin/kafka-log-dirs.sh --bootstrap-server broker1:9092 \
  --describe --topic-list orders

The output is JSON. Pull the sizes out with jq:

bin/kafka-log-dirs.sh --bootstrap-server broker1:9092 \
  --describe --topic-list orders 2>/dev/null | tail -1 | \
  jq '[.brokers[].logDirs[].partitions[]] | map({partition, size}) | sort_by(-.size)'

If one partition of orders is 5x the size of its siblings, your producer keying is the root cause and the fix lives upstream (repartitioning or a compound key), not in the consumer group.

Tool 3: consumer JMX metrics, the trend layer

The CLI gives you snapshots. JMX gives you rates. Every Java consumer exposes these without any agent (Kafka monitoring docs). The MBeans worth scraping, whatever ships them to your metrics store:

MBean group	Metric	What it tells you
`consumer-fetch-manager-metrics`	`records-lag-max`	Worst per-partition lag this consumer sees, your primary alert metric
`consumer-fetch-manager-metrics`	`fetch-rate`, `bytes-consumed-rate`	Whether the consumer is actually pulling; a lag spike with a falling fetch rate points at the consumer, not the producer
`consumer-fetch-manager-metrics`	`records-per-request-avg`	Batch efficiency; chronically low values with high lag suggest fetch tuning, not scaling
`consumer-coordinator-metrics`	`commit-latency-avg`	Slow commits back-pressure the poll loop
`consumer-coordinator-metrics`	`rebalance-rate-per-hour`, `failed-rebalance-rate-per-hour`	Anything persistently above zero in steady state is a churn problem
`consumer-coordinator-metrics`	`last-rebalance-seconds-ago`	A value that keeps resetting to near zero is a rebalance loop, even if each rebalance "succeeds"

The decision rule that saves the most time:

01Consume rate steady

Producer sped up

Lag rises while bytes-consumed-rate holds. Look upstream: a batch job, a replay, new traffic.

02Consume rate falling

Consumer slowed down

Lag rises while bytes-consumed-rate drops. Look at the consumer: a deploy, a slow dependency, a rebalance loop.

One pair of JMX metrics splits every lag incident into two different investigations.

Lag rising while bytes-consumed-rate holds steady means the producer sped up. Lag rising while bytes-consumed-rate drops means the consumer slowed down. Those are different investigations, and this pair of metrics splits them at a glance.

Tool 4: tuning the rebalancing triangle

If Tool 1 showed non-Stable states or Tool 3 showed nonzero rebalance rates, the fix is usually three consumer configs that must be tuned together (consumer configs reference):

# Liveness: how long the coordinator waits for heartbeats before evicting
session.timeout.ms=45000

# Heartbeat cadence: keep at or below one-third of session.timeout.ms
heartbeat.interval.ms=15000

# Processing deadline: max gap between poll() calls before eviction
max.poll.interval.ms=300000

# The lever that actually controls poll-loop duration
max.poll.records=500

People conflate two distinct eviction paths. session.timeout.ms catches dead consumers: heartbeats stop because the background thread died or the network dropped. max.poll.interval.ms catches stuck consumers: heartbeats continue, but poll() never comes back because a batch is taking too long. If your rebalances line up with slow downstream calls, raising session.timeout.ms does nothing. Lower max.poll.records or raise max.poll.interval.ms so one batch reliably fits inside the deadline.

Two more configs turn rebalances from full stops into non-events:

# Incremental cooperative rebalancing: only moved partitions pause
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Static membership: a restart within session.timeout.ms triggers no rebalance at all
group.instance.id=payments-app-pod-0

With the cooperative-sticky assignor, a member joining or leaving revokes only the partitions that actually move, instead of pausing the whole group. Migrating a live group from the default eager assignor takes two rolling restarts (first add CooperativeStickyAssignor alongside the old strategy, then remove the old one), so don't flip it in a single deploy. Static membership via group.instance.id is the best single fix for Kubernetes environments where routine pod restarts were causing a rebalance storm per rollout.

Confluent Cloud: same triage, managed tooling

On Confluent Cloud you don't run broker-side scripts, but every check above has an equivalent.

Console. Go to your cluster → Clients → Consumer lag (or open a topic and use the Consumers tab). You get the same per-partition table (current offset, end offset, lag, member) with sorting, which makes skew-spotting faster than the CLI (Confluent docs).

Confluent CLI. Summarize or list lag for a group directly:

confluent kafka consumer group lag summarize payments-app --cluster lkc-abc123
confluent kafka consumer group lag list payments-app --cluster lkc-abc123

Metrics API. For alerting, scrape confluent_kafka_server_consumer_lag_offsets from the Confluent Metrics API. It's the managed replacement for the JMX layer, computed broker-side, so it works even for non-Java clients.

The Apache scripts still work. kafka-consumer-groups.sh runs fine against Confluent Cloud with an API key in a properties file:

bin/kafka-consumer-groups.sh \
  --bootstrap-server pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 \
  --command-config ccloud.properties \
  --describe --group payments-app

where ccloud.properties contains your sasl.jaas.config with the cluster API key and security.protocol=SASL_SSL.

One lag incident, triaged with native tools only

An illustrative case (names and numbers are made up, the commands are real). At 11:20 someone posts in the team channel: lag on analytics-pipeline has doubled since the morning standup.

The readings. It could be a producer burst (predicts steady consume rate, lag spread across partitions, will drain), a slower consumer (predicts falling consume rate, often after a deploy), a hot partition (predicts lag on one or two partitions only), or a broker problem (predicts URPs). Each one sends you to a different fix, so rule them out in order.

Broker first.

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --under-replicated-partitions

Empty. The brokers are fine, so this is a consumer-side or producer-side problem.

Shape and spread. Run the group describe twice, 60 seconds apart. Lag grew on all 12 partitions, roughly evenly, and LOG-END-OFFSET advanced at the usual pace. Not a hot key, not a producer surge.

Rate. The consumer dashboard shows bytes-consumed-rate down about 40% since 09:50, and rebalance-rate-per-hour at zero. The consumer slowed down, and it isn't churning.

Change. The deploy log for the consumer service shows a release at 09:44 that added an enrichment call per record. That lines up.

Verdict. A consumer regression from the 09:44 release, steady growth shape. The options are a rollback or more instances (up to the 12-partition ceiling). Then the last check, retention: lag age is about 3 hours on a 72-hour topic, so there's time to roll back calmly instead of scaling in a panic. Elapsed time for someone fluent in the tools: 25 to 40 minutes, most of it spent finding the deploy.

What each native tool answers, and what it doesn't

Tool	Answers	Doesn't answer
`kafka-consumer-groups.sh`	Lag per partition, assignment, group state	Why it changed, or when the trend broke
`kafka-topics.sh`, `kafka-log-dirs.sh`	Replication health, partition sizes	Whether the broker issue explains this lag
JMX consumer metrics	Rates and rebalance frequency over time	Which deploy or config change moved them
Timeout and assignor configs	Stops eviction loops and restart storms	Whether this incident is an eviction loop
Confluent Cloud console, CLI, Metrics API	The same views, managed	The same gaps

Every row on the right is the correlation step: lining lag up against deploys, config changes and producer traffic. That's the step the worked example spent most of its time on.

Where the native-tools approach runs out

Everything above is worth doing, and none of it needs a purchase order. But after a few incidents you'll notice the same four walls:

01Wall 1

Snapshots

LAG=180000 says nothing about whether a deploy, a burst or a rebalance loop caused it.

02Wall 2

Polling

A cron job sees lag on its schedule. Bursts that drain between runs never show up.

03Wall 3

JMX pipelines

Exporters, dashboards and per-group thresholds are a project, and they go stale.

04Wall 4

Diagnosis only

Knowing the fix still leaves a config change, a review and a rolling restart.

Where native tooling stops. The first wall is the one that costs you at 2 a.m.

Scripts snapshot lag; they don't explain it. LAG=180000 says nothing about whether a deploy slowed the consumer, a producer burst outpaced it, or a rebalance loop kept resetting progress. The correlation work (timelines, deploy history, config diffs) is still yours, at 2 a.m.

Polling isn't watching. A cron job around kafka-consumer-groups.sh catches lag on its schedule. Bursty lag that appears and drains between runs, often the early symptom of the incident that eventually sticks, never shows up.

JMX pipelines are their own project. Exporters, dashboards and alert thresholds per group and per topic take real engineering time, and thresholds go stale as traffic grows.

Diagnosis isn't remediation. Knowing that max.poll.records is too high for the new payload size still leaves a config change, a review and a rolling restart between you and a drained backlog.

What comes next

Every check in this guide can run continuously instead of when someone remembers. CloudThinker connects to Kafka or Confluent Cloud read-only, watches lag, rebalance rates, skew and replication health around the clock, and when lag grows it does the correlation work (deploy, rebalance loop or hot partition) and proposes the fix for your approval. That's part three of this series.