Elasticsearch Index Management: 6 Problems Hiding in Your Cluster

Your Elasticsearch cluster went yellow six months ago. Someone checked, said "it's just a replica," and closed the tab. The dashboard tile has been amber ever since, and at this point it would be more alarming if it turned green.

That normalized yellow is the signature failure of Elasticsearch index management. The cluster keeps answering queries, so nobody treats degradation as a problem until search latency doubles, or a node hits the flood-stage watermark and indexing stops at 3 p.m. on a Tuesday. Between "working" and "down" there's a long, expensive middle where the cluster is slower and larger than it needs to be. Almost every symptom in that middle traces back to how indices are created, sized and retired.

This guide covers the six problems we see most on self-managed and Elastic Cloud clusters alike: what each one is, why it happens, one API call to detect it, and what it typically costs in latency and infrastructure.

This is part one of a three-part series. Part two is a hands-on tour of index management with native Elasticsearch tools: the _cat APIs, ILM policies, slow logs. Part three covers automating the whole loop with an AI agent.

01Redundancy

Yellow forever

Replicas unassigned for weeks, so one node failure becomes data loss.

02Geometry

Oversharding

Thousands of tiny shards, each paying heap, coordination and cluster-state cost.

03Retention

No rollover

ILM missing or stuck, so one index grows forever on the hot tier.

04Evidence

No slow logs

Thresholds never set, so the slow query is gone before anyone looks.

05Capacity

Watermarks

Disks past 85% stop allocation; at 95% writes to affected indices fail.

06Schema

Mapping explosion

Dynamic keys become fields until indexing hits the total fields limit.

The six problems this guide covers, each with one API call that detects it.

All commands assume localhost:9200. Swap in your endpoint and add -u user:pass or -H "Authorization: ApiKey ..." as needed.

Why a status color can't manage a cluster

Most teams watch Elasticsearch through two things: the green/yellow/red health status and a handful of threshold alerts (heap above 85%, disk above 80%). Both are structurally blind to the problems below.

Health status is a summary of allocation, not of fitness. A cluster with 30,000 one-megabyte shards, a 2 TB index that never rolled over, and no slow logs can be perfectly green. Green only says every shard copy found a home.

Threshold alerts fire on symptoms, one metric at a time. The disk alert fires on the node that filled up. It doesn't know the disk filled because an ILM policy was never attached, so the fix that ships is "add a bigger disk," and the cause keeps growing.

The cluster tolerates neglect for a long time. Elasticsearch degrades gracefully right up to the point where it doesn't. Nothing errors when a rollover condition can never trigger. Nothing errors when a mapping gains its 900th field. The feedback arrives months later, as an incident.

The knowledge lives in someone's head. Which index patterns belong to which service, which retention the compliance team actually signed off on, why replicas were set to 2 on that one index: that context rarely makes it into the cluster or the runbook.

So the checks below are point lookups you have to go and make. None of them fire by themselves.

1. Cluster health stuck in yellow

What it is. Yellow means every primary shard is assigned but at least one replica isn't. Queries still work, which is exactly why teams stop caring. But a yellow cluster has no redundancy for the affected indices: lose the node holding those primaries and you lose data. Yellow is also often a symptom of something else on this list, like a disk watermark blocking allocation or more shards than the nodes can hold.

Why it happens. Replica counts higher than the node count can satisfy (2 replicas on a 2-node cluster will never go green), a node that left and never came back, or allocation rules that quietly exclude the only eligible node. Nobody investigates because "it still works."

How to detect it.

curl -s "localhost:9200/_cluster/health?pretty"

Look past status. unassigned_shards tells you the size of the problem. initializing_shards stuck above zero for hours means allocation is wedged, not in progress, and delayed_unassigned_shards above zero means Elasticsearch is deliberately waiting for a departed node to come back (the default index.unassigned.node_left.delayed_timeout is one minute). When you're ready to find out why a shard is unassigned, the cluster allocation explain API gives you the exact reason. We walk through it in part two.

Typical impact. Zero, until a node fails. Then it's data loss on every index whose only copy lived there. Treat sustained yellow as an incident with a slow fuse.

2. Oversharding and tiny-shard sprawl

What it is. Thousands of shards holding a few megabytes each. Every shard is a full Lucene index with its own segment files, memory overhead, and entry in the cluster state. A search across an oversharded index fans out to every shard and pays coordination cost on each, and the master re-publishes a bloated cluster state on every change. Elastic's own guidance is to aim for shards between 10 GB and 50 GB. Most sprawled clusters average under 1 GB.

Why it happens. Daily indices created per service, per environment, per customer, each with the pre-7.0 default of 5 primaries and 1 replica. Ten services × 365 days × 10 shards is 36,500 shards a year, most of them nearly empty.

How to detect it.

curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,store,node&s=store:asc" | head -50

Sorting by store size ascending puts the junk on top. If the first screen is full of shards under 100 MB, you're oversharded. For a total, _cluster/health returns active_shards; divide your data volume by it to get the average. The older rule of thumb (stay well under 20 shards per GB of heap per node) is still a useful smell test on pre-8.3 clusters.

Typical impact. This is the big one for both latency and cost. Consolidating tiny daily indices into rollover-managed ones commonly cuts shard counts by 80 to 90%, drops search p99 noticeably on multi-index queries, and frees enough heap to remove a node or two, often 15 to 30% of cluster infrastructure cost.

3. Indices that never roll over

What it is. An index lifecycle management (ILM) policy that exists but was never attached, or was attached with a rollover action that can never trigger, usually because the index wasn't created through an alias or data stream, so there's nothing to roll over. The index grows forever: 800 GB, one shard, no delete phase ever firing. The cluster stores every log line since 2023 on hot-tier NVMe.

Why it happens. ILM has real setup requirements: an index template, a write alias or data stream, and the policy. If any leg is missing, ILM either fails with an error nobody reads or doesn't run at all. The index keeps accepting writes, so the failure is invisible until the disk isn't.

How to detect it.

# Indices where ILM is erroring
curl -s "localhost:9200/logs-*/_ilm/explain?only_errors=true&pretty"

# Indices with no policy at all
curl -s "localhost:9200/logs-*/_ilm/explain?pretty" \
  | jq -r '.indices[] | select(.managed == false) | .index'

An empty first result isn't a clean bill of health. The second call catches indices that ILM has never heard of, and those are the ones that grow without bound. Also scan the full output for indices sitting in the same action and step for weeks. The ILM documentation covers what each phase and step looks like when healthy.

Typical impact. Retention is your single biggest storage lever. Clusters that fix broken ILM typically discover they're storing 2 to 5× the data their stated retention policy implies. On hot/warm architectures, that's pure hot-tier spend.

4. Slow logs nobody enabled

What it is. Elasticsearch ships with search and indexing slow logs: per-index logs that capture any query or indexing operation exceeding thresholds you set. By default no thresholds are set, so nothing is logged. When search "feels slow," teams have zero evidence about which queries, which indices or which patterns are responsible, and the investigation starts from guesswork.

Why it happens. Slow logs are opt-in and per index, so they never get enabled proactively. They get enabled mid-incident, after the offending query has already run.

How to detect it. Check whether any index has thresholds configured:

curl -s "localhost:9200/_all/_settings/index.search.slowlog*,index.indexing.slowlog*?pretty"

If every index comes back with an empty settings object, you're flying blind. Enabling them is one settings call with thresholds like index.search.slowlog.threshold.query.warn: 2s. Exact syntax and how to read the output are in the slow log docs and in part two of this series.

Typical impact. Indirect but large: the difference between a two-hour incident ("we can see the exact wildcard query from the reporting service") and a two-day one. Elasticsearch slow queries are almost always a handful of repeat offenders (leading-wildcard searches, huge aggregations, deep pagination), and slow logs name them immediately.

5. Disk watermark pressure and flood-stage lockout

What it is. Elasticsearch protects itself with three disk thresholds (disk-based shard allocation). At the low watermark (85% by default) it stops allocating new shards to the node. At the high watermark (90%) it actively relocates shards away. At flood stage (95%) it puts index.blocks.read_only_allow_delete on every index with a shard on that node, and writes to those indices start failing. Ingest pipelines back up, apps throw 429s, and the on-call gets paged for what is really a retention problem.

Why it happens. Usually problem 3 (indices that never roll over or delete) meets steadily growing log volume. Watermark breaches are the symptom that finally makes broken ILM visible.

How to detect it.

curl -s "localhost:9200/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail,shards&s=disk.percent:desc"

Any node above 85% is already distorting shard placement. Above 90%, it's moving data around under load. Check this weekly, not after the page.

Typical impact. Flood stage is a production outage for ingest. Even below it, high-watermark relocations consume disk and network I/O that competes directly with search, so clusters running hot on disk routinely see elevated latency for hours during rebalancing.

6. Mapping explosions from dynamic fields

What it is. With dynamic mapping on (the default), every new JSON key in an ingested document becomes a new field in the index mapping. Log payloads that use user IDs, request IDs or arbitrary labels as keys (a fresh user_12345 object key in every document) generate a new field per document. Mappings balloon, the cluster state grows with them, and eventually indexing fails against index.mapping.total_fields.limit (default 1,000).

Why it happens. Some upstream service serializes a map with dynamic keys, and nobody notices until indexing breaks or the master starts struggling to publish cluster state. Raising the field limit "to fix it" makes the explosion legal, not gone.

How to detect it. Count fields on a suspect index:

curl -s "localhost:9200/logs-app-000001/_field_caps?fields=*" | jq '.fields | length'

Anything drifting toward four digits on a logging index deserves a look at the mapping itself (GET logs-app-000001/_mapping) to find the exploding branch. The fix is dynamic: false or a flattened field on the offending object. See mapping limit settings.

Typical impact. Slower cluster-state publishing, heap pressure on every node, and hard indexing failures when the limit hits. On busy logging clusters, one misbehaving service can take ingest down for everyone sharing the index template.

The pattern: these problems create each other

Read the six together and a chain shows up.

01Root

ILM detached

The index was created outside the template, so no policy ever runs.

02Growth

Index balloons

One shard grows past a terabyte on the hot tier.

03Disk

Low watermark

The node crosses 85% and stops accepting new shards.

04Symptom

Yellow

Replicas cannot allocate. The dashboard turns amber and stays there.

Four of the six problems are often one chain. The dashboard only shows the last step.

ILM that never attaches (3) grows an index until a disk crosses a watermark (5). That blocks allocation and strands replicas, which parks the cluster in yellow (1). Everyone ignores it because search still works, only slower, because one index has a giant shard while the rest of the cluster drowns in tiny ones (2), and there are no slow logs (4) to prove any of it.

Here's what that looks like when you walk it backwards on a real cluster. Say _cluster/health reports yellow with 42 unassigned shards. Three readings fit:

A node left. Then unassigned_info.reason is NODE_LEFT and the shards come back once the node does.
Replicas can't fit. Then the allocation explain output shows the same_shard decider on every node, and number_of_replicas is at least the data node count.
Disk is full. Then the disk_threshold decider says no on the target node, and _cat/allocation shows it above 85%.

One call separates them:

curl -s -X POST "localhost:9200/_cluster/allocation/explain?pretty"

If it's the third reading, the question becomes why that node filled, and _cat/indices?s=store.size:desc usually names one index far past any sane rollover size. Its _ilm/explain then shows "managed": false or a step stuck for weeks. That's five calls across four APIs to get from an amber tile to a cause, and none of them run unless someone decides to run them.

That's why point-in-time checks age so badly here. Each command takes seconds, but the answers change weekly: every new service adds index patterns, every traffic bump changes the rollover math, every schema change can start a mapping explosion. Cluster hygiene isn't a project you finish. It's a rate you have to keep up with.

Where this series goes next

Run all six checks today. Most teams find at least three problems on the first pass. Then read part two, the full DIY version with native tools: _cat API workflows, writing and attaching an ILM policy that actually rolls over, enabling slow logs, and debugging unassigned shards with allocation explain.

Part three makes the checks continuous. CloudThinker connects to Elasticsearch with a read-only API key, watches cluster health, shard geometry, ILM progress and snapshots, and walks chains like the one above for you, so the finding arrives as "this index, this policy, this fix" instead of six separate graphs. It acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Try CloudThinker free with 100 premium credits and no card required, or continue with the hands-on native-tools guide.