Hands-On Elasticsearch Index Management with Native Tools

Search p95 on the logging cluster doubled overnight. The Kibana tile says yellow, which it's said for months. Heap looks fine on the overview. You have no slow logs, no idea which index the slow queries hit, and a product team asking whether it's "an Elasticsearch thing."

Five _cat calls would already tell you more than that dashboard: _cat/health, _cat/indices, _cat/shards, _cat/allocation and _cat/nodes. They're built in, they cost nothing, and they answer the questions dashboards blur: which index is eating the disk, which shard never got assigned, which node is running hot. This guide is a working tour of Elasticsearch index management using only what ships with the cluster: the _cat APIs, allocation explain, index lifecycle management (ILM), index templates, slow logs and snapshot lifecycle management (SLM). Every command is copy-pasteable curl.

In part one of this series we walked through the six signals that matter: yellow cluster health, oversharding, indices that never roll over, slow logs nobody enabled, disk watermark pressure and mapping explosions. This is part two, the hands-on cleanup. Part three covers running these checks continuously with an AI agent.

Build in this order. Each tool answers a question the next one depends on.

01Look

_cat APIs

Health, indices, shards, disk and nodes in five calls.

02Explain

Allocation explain

The decider that kept each unassigned shard off each node.

03Retain

ILM and templates

Rollover and delete on a policy, wired in through a data stream.

04Evidence

Slow logs

The exact query or indexing call that crossed your threshold.

05Recover

SLM

Scheduled snapshots, so retention deletes stay reversible.

All examples assume localhost:9200. Substitute your endpoint and auth (-u user:pass or -H "Authorization: ApiKey ...") as needed.

---

Tool 1: the _cat APIs, ground truth in five calls

The _cat APIs return human-readable tables. Two parameters do most of the work: h= picks columns and s= sorts. Add v for headers.

# 1. Cluster at a glance: status, node count, unassigned shards
curl -s "localhost:9200/_cat/health?v"

# 2. Indices, biggest first: where the disk went
curl -s "localhost:9200/_cat/indices?v&h=index,health,pri,rep,docs.count,store.size,creation.date.string&s=store.size:desc"

# 3. Shards, smallest first: the tiny-shard sprawl hunt
curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=store:asc"

# 4. Disk per node: who's approaching the watermark
curl -s "localhost:9200/_cat/allocation?v&s=disk.percent:desc"

# 5. Node hot spots: heap, CPU, load
curl -s "localhost:9200/_cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent&s=heap.percent:desc"

How to read them. In call 2, sort by creation.date.string on a second pass. Daily indices from eight months ago that were supposed to be deleted at 30 days are the single most common finding. In call 3, any log index shard under roughly 1 GB is sprawl (target 10 to 50 GB per shard), and anything in state UNASSIGNED goes straight to Tool 2. In call 4, disk.percent above 85 means the node is past the low watermark and Elasticsearch has already stopped allocating new shards to it. In call 5, sustained heap above 85% is where circuit-breaker errors start.

The _cat output is for your eyes. When you script these checks in cron, use the JSON APIs (_cluster/health, _stats, _nodes/stats) instead. Elastic explicitly recommends against parsing _cat text in automation.

Tool 2: allocation explain, why that shard is unassigned

Yellow or red health always comes down to shards that couldn't be placed, and _cluster/allocation/explain gives you the actual reason instead of a guess:

# Explain the first unassigned shard Elasticsearch finds
curl -s -X POST "localhost:9200/_cluster/allocation/explain?pretty"

# Or ask about a specific shard copy
curl -s -X POST "localhost:9200/_cluster/allocation/explain?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "index": ".ds-logs-app-default-2026.07.01-000042",
  "shard": 0,
  "primary": false
}'

Skip to allocate_explanation, unassigned_info.reason and the per-node node_allocation_decisions[].deciders array. The text is verbose but honest:

• disk_threshold says no: that node is past a watermark. Go back to _cat/allocation.
• same_shard says no on every node: there are more copies than nodes, which on a single-node cluster means number_of_replicas should be 0.
• unassigned_info.reason: NODE_LEFT with a delay: recovery is already scheduled for when the node returns. Don't touch anything.
• filter says no: an allocation filter (index.routing.allocation.*) excludes the only eligible nodes, often a leftover from a migration.

The decider tells you whether the fix is disk, settings or patience.

Tool 3: ILM, rollover and delete on autopilot

Index lifecycle management is how indices stop growing forever. A minimal, production-sane policy for logs: roll over the write index when a primary shard hits 50 GB or the index is 7 days old, and delete 30 days after rollover.

curl -s -X PUT "localhost:9200/_ilm/policy/logs-30d" \
  -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}'

A policy does nothing until an index uses it, and the clean way to wire that up is an index template. For logs, use a data stream. It handles the write alias and backing-index naming for you:

curl -s -X PUT "localhost:9200/_index_template/logs-app" \
  -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-30d"
    }
  }
}'

The priority of 200 matters: it beats the built-in logs-*-* template (priority 100) that would otherwise match the same names. From now on, anything written to logs-app-default lands in a managed data stream: one primary shard per backing index, rollover at 50 GB or 7 days, deletion around 37 days. ILM checks conditions on a poll interval (indices.lifecycle.poll_interval, default 10 minutes), so rollover isn't instant, and that's fine.

Templates only apply at index creation. Existing indices keep whatever they had, which is exactly how part one's "never rolled over" index came to exist.

Debugging stuck indices with _ilm/explain

The most common ILM failure isn't an error. It's an index that quietly never advances. Ask Elasticsearch directly:

# Everything that's erroring
curl -s "localhost:9200/logs-app-*/_ilm/explain?only_errors=true&pretty"

# Full lifecycle state for one backing index
curl -s "localhost:9200/.ds-logs-app-default-2026.07.01-000042/_ilm/explain?pretty"

Read three fields: phase, action and step. A healthy waiting index shows "step": "check-rollover-ready". If step is ERROR, the step_info object tells you why. The classic on a non-data-stream setup is index.lifecycle.rollover_alias [...] does not point to index [...], which means the alias bootstrap was skipped or the alias was repointed during a migration. After fixing the cause, retry the step:

curl -s -X POST "localhost:9200/logs-app-000042/_ilm/retry"

If _ilm/explain returns "managed": false, the policy was never attached at all: the template existed but the index was created before it, or the pattern doesn't match. That's the "never rolled over" scenario from part one, and it's why a 2 TB index exists.

Tool 4: slow logs, see the queries your p95 hides

Slow logs are off by default and configured per index (docs). Enable search and indexing thresholds on the indices that matter:

curl -s -X PUT "localhost:9200/logs-app-*/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "1s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "5s",
  "index.indexing.slowlog.threshold.index.info": "1s",
  "index.indexing.slowlog.source": "1000"
}'

To make these stick for future backing indices, put the same settings in the index template from Tool 3. Settings applied to a wildcard only reach indices that exist today.

Entries land in JSON log files next to the main server log (*_index_search_slowlog.json, *_index_indexing_slowlog.json). On Elastic Cloud, enable logs and metrics shipping to reach them. Each search entry includes took, the shard and the full query source, which is the point. When someone says "search is slow," this file tells you which query.

Walking the opening example

Back to the overnight p95 doubling. Three readings fit, and each predicts something different:

01Reading 1

Cluster problem

Many distinct slow queries across many indices. Heap is high or shards are tiny.

02Reading 2

Query problem

One query shape repeats in the slow log: a leading wildcard, deep paging, a huge aggregation.

03Reading 3

Rebalancing

Slowness tracks a node past the high watermark, with shards relocating during business hours.

Check them in order of cost:

1. Rebalancing. _cat/allocation?s=disk.percent:desc and _cat/recovery?active_only=true&v. A node above 90% with active relocations during the slow window points here, and the fix is retention, not queries.
2. Cluster problem. _cat/nodes for heap, and _cluster/health for active_shards. Heap above 85% on several nodes plus thousands of sub-gigabyte shards points at sprawl. The slow log will show many different queries, all slow.
3. Query problem. Group the slow log by query shape. The same leading-wildcard query_string on message showing up 4,000 times since yesterday's reporting release is a query problem, and the fix belongs to the team that shipped it.

The tools answer each question cleanly. Deciding which question to ask next is still you.

Tool 5: SLM, snapshots that actually happen

Retention via ILM delete phases is only safe if you can restore what you deleted by mistake. Snapshot lifecycle management automates backups to a registered repository:

# Register a repository (S3 shown; also fs, azure, gcs)
curl -s -X PUT "localhost:9200/_snapshot/backups" \
  -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": { "bucket": "my-es-snapshots" }
}'

# Nightly snapshot at 01:30, keep 30 days, between 5 and 50 copies
curl -s -X PUT "localhost:9200/_slm/policy/nightly" \
  -H 'Content-Type: application/json' -d'
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "backups",
  "config": { "indices": ["*"], "include_global_state": false },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}'

# Run it once now, then check the scoreboard
curl -s -X POST "localhost:9200/_slm/policy/nightly/_execute"
curl -s "localhost:9200/_slm/stats?pretty"

_slm/stats is the check people skip. A failure count climbing quietly is how teams discover, mid-incident, that the last good snapshot is six weeks old. GET _slm/policy/nightly shows last_success and last_failure per policy. Put that call in a weekly review.

---

What native tools cover, and what they leave

Down the right-hand column it's the same story: the cluster records the facts, and a person has to read them, connect them and act.

The honest part: where the DIY loop stops

Run everything above and you'll have a healthier, cheaper cluster. This pass typically claws back 20 to 40% of storage on a cluster that's never had ILM discipline. But know what you've built.

01Correlation

Yours to do

Each API answers one question. Linking yellow to a watermark to a stuck policy is still a person.

02Cadence

Point in time

Shard sprawl, stuck ILM steps and failed snapshots come back between passes.

03Fixes

Tickets

Slow logs name the query and ILM names the step. Nobody is assigned to act on either.

It observes; it doesn't investigate. A cron job can tell you the cluster went yellow at 3 a.m. It can't connect the yellow to the unassigned shard, the shard to the watermark, and the watermark to the one index whose ILM policy silently detached in March. That correlation is still an engineer with six terminal tabs open.

Findings decay between passes. Shard sprawl, stuck ILM steps and failed snapshots regenerate continuously. The audit you ran this sprint describes last sprint's cluster.

Every fix is still manual. Slow logs name the bad query; nobody rewrites it. _ilm/explain names the stuck step; nobody retries it. Detection produces tickets, and tickets queue.

What comes next

Everything in this guide can run continuously instead of quarterly. In part three, CloudThinker connects to Elasticsearch with a read-only API key, watches cluster health, shard geometry, ILM progress and snapshots, and when something degrades it walks the chain from symptom to cause and proposes the fix. Writes like a rollover or a policy change go through Auto Mode, so you decide what runs on its own and what waits for approval.

If you want to see it against your own cluster first, ask something like this in your first session:

"Search p95 on the logging cluster doubled since yesterday. Check disk and relocations, heap and shard counts, and the heaviest indices, and tell me which reading the evidence supports. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with part three.