Hands-On Elasticsearch Index Management with Native Tools
Five _cat API calls will tell you more about your cluster's health than most dashboards: _cat/health, _cat/indices, _cat/shards, _cat/allocation, and _cat/nodes. They're built in, they cost nothing, and they answer the questions dashboards tend to blur — which index is eating the disk, which shard never got assigned, which node is running hot. This guide is a working tour of Elasticsearch index management using only what ships with the cluster: the _cat APIs, index lifecycle management (ILM), index templates, slow logs, allocation explain, and snapshot lifecycle management. Every command is copy-pasteable curl.
In part one of this series we walked through the six signals that matter — yellow cluster health, oversharding, indices that never roll over, slow logs nobody enabled, disk watermark pressure, and mapping explosions. This is part two: the hands-on cleanup. Part three covers running these checks continuously with an AI agent.
All examples assume localhost:9200; substitute your endpoint and auth (-u user:pass or -H "Authorization: ApiKey ...") as needed.
Tool 1: the _cat APIs — ground truth in five calls
The _cat APIs return human-readable tables. Two flags do most of the work: h= picks columns, s= sorts. Add v for headers.
# 1. Cluster at a glance: status, node count, unassigned shards
curl -s "localhost:9200/_cat/health?v"
# 2. Indices, biggest first — where the disk went
curl -s "localhost:9200/_cat/indices?v&h=index,health,pri,rep,docs.count,store.size,creation.date.string&s=store.size:desc"
# 3. Shards, smallest first — the tiny-shard sprawl hunt
curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=store:asc"
# 4. Disk per node — who's approaching the watermark
curl -s "localhost:9200/_cat/allocation?v&s=disk.percent:desc"
# 5. Node hot spots — heap, CPU, load
curl -s "localhost:9200/_cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent&s=heap.percent:desc"
How to read them. In call 2, sort by creation.date.string on a second pass — daily indices from eight months ago that were supposed to be deleted at 30 days are the single most common finding. In call 3, any log index shard under roughly 1 GB is sprawl (target 10–50 GB per shard); anything in state UNASSIGNED goes straight to the allocation-explain section below. In call 4, disk.percent above 85 means you're at the low watermark and Elasticsearch has already stopped allocating new shards to that node. In call 5, sustained heap above 85% is where circuit-breaker errors start.
The _cat output is for your eyes; when you script these checks in cron, use the JSON equivalents (_cluster/health, _stats) instead — Elastic explicitly recommends against parsing _cat text in automation.
Tool 2: ILM — rollover and delete on autopilot
Index lifecycle management is how indices stop growing forever. A minimal, production-sane policy for logs: roll over the write index when it hits 50 GB per primary shard or 7 days, delete 30 days after rollover.
curl -s -X PUT "localhost:9200/_ilm/policy/logs-30d" \
-H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_primary_shard_size": "50gb",
"max_age": "7d"
}
}
},
"delete": {
"min_age": "30d",
"actions": { "delete": {} }
}
}
}
}'
A policy does nothing until an index uses it, and the clean way to wire that up is an index template. For logs, use a data stream — it handles the write alias and index naming for you:
curl -s -X PUT "localhost:9200/_index_template/logs-app" \
-H 'Content-Type: application/json' -d'
{
"index_patterns": ["logs-app-*"],
"data_stream": {},
"priority": 200,
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "logs-30d"
}
}
}'
From now on, anything written to logs-app-default creates a managed data stream: one primary shard per backing index, rollover at 50 GB or 7 days, gone at 37 days. Note that ILM checks conditions on a poll interval (indices.lifecycle.poll_interval, default 10 minutes) — rollover is not instant, and that's fine.
Debugging stuck indices with _ilm/explain
The most common ILM failure isn't an error — it's an index that quietly never advances. Ask Elasticsearch directly:
# Everything that's erroring
curl -s "localhost:9200/logs-app-*/_ilm/explain?only_errors&pretty"
# Full lifecycle state for one index
curl -s "localhost:9200/.ds-logs-app-default-2026.07.01-000042/_ilm/explain?pretty"
Read three fields: phase, action, and step. A healthy waiting index shows "step": "check-rollover-ready". If step is ERROR, the step_info object tells you why — the classic being index.lifecycle.rollover_alias does not point to index, which means the alias bootstrap was skipped on a non-data-stream setup. After fixing the cause, retry the step:
curl -s -X POST "localhost:9200/logs-app-000042/_ilm/retry"
If _ilm/explain returns "managed": false, the policy was never attached at all — the template existed but the index was created before it, or the pattern doesn't match. That's the "never rolled over" scenario from part one, and it's why a 2 TB index exists.
Tool 3: slow logs — see the queries your p95 hides
Slow logs are off by default and configured per index (docs). Enable both search and indexing thresholds on the indices that matter:
curl -s -X PUT "localhost:9200/logs-app-*/_settings" \
-H 'Content-Type: application/json' -d'
{
"index.search.slowlog.threshold.query.warn": "5s",
"index.search.slowlog.threshold.query.info": "1s",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.indexing.slowlog.threshold.index.warn": "5s",
"index.indexing.slowlog.threshold.index.info": "1s",
"index.indexing.slowlog.source": "1000"
}'
Entries land in JSON log files next to the main server log (*_index_search_slowlog.json, *_index_indexing_slowlog.json). Each search entry includes took, the shard, and the full query source — which is the point. When someone says "search is slow," this file tells you which query: the wildcard-prefix query, the aggregation across 400 fields, the query fanning out to 900 tiny shards.
How to read it. Two patterns dominate. Many distinct slow queries across many indices usually means a cluster problem — oversharding or heap pressure — so go back to tools 1 and 2. The same query shape repeatedly means a query problem: leading wildcards, huge size values, or scripted sorting. Set info thresholds around your SLO (a 1-second search on a log cluster is worth knowing about) and warn at the level that pages a human. Start conservative; slow logging is cheap but not free.
Tool 4: allocation explain — why that shard is unassigned
Yellow or red health always comes down to shards that couldn't be placed, and _cluster/allocation/explain gives you the actual reason instead of a guess:
# Explain the first unassigned shard Elasticsearch finds
curl -s -X POST "localhost:9200/_cluster/allocation/explain?pretty"
# Or ask about a specific shard
curl -s -X POST "localhost:9200/_cluster/allocation/explain?pretty" \
-H 'Content-Type: application/json' -d'
{
"index": "logs-app-000042",
"shard": 0,
"primary": false
}'
Skip to can_allocate and the per-node deciders array. The explanations are verbose but honest: disk threshold means a node is past the watermark (see _cat/allocation above), same shard already allocated to this node on a single-node cluster means your replica count should be 0, and node left the cluster with a delay timer means recovery is already scheduled — don't touch anything. The decider text tells you whether the fix is disk, settings, or patience.
Tool 5: SLM — snapshots that actually happen
Retention via ILM delete phases only works if you can restore what you deleted by mistake. Snapshot lifecycle management automates backups against a registered repository:
# Register a repository (S3 shown; also fs, azure, gcs)
curl -s -X PUT "localhost:9200/_snapshot/backups" \
-H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": { "bucket": "my-es-snapshots" }
}'
# Nightly snapshot at 01:30, keep 30 days, 5-50 copies
curl -s -X PUT "localhost:9200/_slm/policy/nightly" \
-H 'Content-Type: application/json' -d'
{
"schedule": "0 30 1 * * ?",
"name": "<nightly-{now/d}>",
"repository": "backups",
"config": { "indices": ["*"], "include_global_state": false },
"retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}'
# Run it once now, then check the scoreboard
curl -s -X POST "localhost:9200/_slm/policy/nightly/_execute"
curl -s "localhost:9200/_slm/stats?pretty"
_slm/stats is the check people skip: snapshots_failed climbing silently is how teams discover, mid-incident, that the last good snapshot is six weeks old. Put that one call in a weekly review.
The honest part: where the DIY loop stops
Run everything above and you'll have a healthier, cheaper cluster — this pass typically claws back 20–40% of storage on a cluster that's never had ILM discipline. But know what you've actually built:
It observes; it doesn't investigate. A cron job can tell you the cluster went yellow at 3 a.m. It cannot connect the yellow status to the unassigned shard, the unassigned shard to the disk watermark, and the watermark to the one index whose ILM policy silently detached in March. That correlation is still an engineer with six terminal tabs open.
Findings decay between passes. Shard sprawl, stuck ILM steps, and failed snapshots regenerate continuously. The audit you ran this sprint describes last sprint's cluster.
Every fix is still manual. Slow logs name the bad query; nobody rewrites it. _ilm/explain names the stuck step; nobody retries it. Detection produces tickets, and tickets queue.
What comes next
Everything in this guide can run continuously instead of quarterly. CloudThinker agents connect to Elasticsearch with a read-only API key and watch cluster health, shard balance, ILM progress, and slow logs around the clock — and when something degrades, they investigate the chain and propose the fix, gated behind your approval. That's part three of this series.
Or see your own cluster's findings first: Try CloudThinker free — 100 premium credits, no card required.