Automating Elasticsearch Index Management with AI Agents

There's a real difference between a Slack alert that says cluster status: yellow and a message that says: the cluster went yellow at 03:12 because 42 replica shards can't allocate. Node es-data-4 crossed the low disk watermark, and the disk filled because logs-app-000001 has grown to 1.4 TB without ever rolling over. Its ILM policy is attached but stuck on check-rollover-ready, because the rollover alias points at a different index. Here's the fix; approve to apply.

The first message is a fact. The second is an investigation. Most teams running Elasticsearch have plenty of the first kind and produce the second kind by hand, at whatever hour the page arrives.

This is part three of our Elasticsearch index management series. Part one mapped the signals that matter: yellow-forever clusters, oversharding, stuck ILM, slow logs nobody enabled, disk watermarks and mapping explosions. Part two covered the DIY toolkit: _cat APIs, ILM policies, slow logs, allocation explain and snapshot lifecycle management. If you followed along, you can diagnose everything in that opening paragraph yourself. The catch is that you have to keep doing it, every incident and every quarter.

This part covers the alternative: CloudThinker as a continuous action layer on top of your cluster, doing the watching and the correlating, and proposing the fixing for you to approve. Your cluster, your Kibana and your index templates stay where they are.

---

Why scripts and alerts stop at the symptom

By the end of part two, most teams have some version of the same setup: a cron job calling _cluster/health, a disk alert at 80%, maybe a Watcher or Kibana rule on heap. It's worth having. It also has a structural ceiling.

Each check answers one question. The health check knows the cluster is yellow. The disk alert knows es-data-4 is at 87%. Neither knows the other fired, and neither knows that one index whose ILM policy detached in March is the reason for both.

The next question depends on the last answer. Yellow can mean a node left, replicas that can't fit, or a full disk. Each answer sends you to a different API. A script can only follow the branch someone wrote in advance, and the branch nobody wrote is the incident you're having.

The thresholds are static and the cluster isn't. A shard count that was fine at 2 TB is sprawl at 8 TB. A 1-second slow log threshold that captured ten queries a day captures ten thousand after a reporting release.

The context lives in heads. Which team owns metrics-billing-*, whether 30-day retention is a compliance number or a guess, why replicas are 2 on one index: none of that is in the cluster, so an alert can't use it.

An agent doesn't replace those checks. It does the part after them: read the evidence, decide what to look at next, and write the reasoning down.

---

Connecting: a read-only API key, about five minutes

01Cluster

Elasticsearch

• Health and allocation_cluster/health, allocation explain
• Indices, shards, nodes_cat and _stats APIs
• ILM and SLM stateread_ilm, read_slm

Monitor-only API key

02Investigation

CloudThinker

• Watch continuouslyHealth, geometry, lifecycle, snapshots
• Walk the chainSymptom to decider to index to policy
• Propose the fixExact API calls, with rollback notes

No plugin on the cluster

03Where the team works

Chat and tickets

• Slack or TeamsFindings and approvals in the thread
• PagerDutySame escalation path, evidence attached
• JiraOwned follow-ups for query and mapping fixes

Nothing new to watch

CloudThinker connects to Elasticsearch the way your security review would demand anyway: an API key scoped to monitoring privileges, no write access, no plugin installed on the cluster. You create the key yourself, so you can see exactly what's granted:

POST /_security/api_key
{
  "name": "cloudthinker-readonly",
  "role_descriptors": {
    "cloudthinker_monitor": {
      "cluster": ["monitor", "read_ilm", "read_slm"],
      "indices": [
        {
          "names": ["*"],
          "privileges": ["monitor", "view_index_metadata"]
        }
      ]
    }
  }
}

monitor covers cluster health, node stats, _cat and allocation explain. read_ilm and read_slm cover lifecycle and snapshot policy state. view_index_metadata covers mappings and settings, which is what mapping-explosion checks need. None of these privileges can read a document. If you ship slow logs into an index on the cluster and want them grouped by query shape, add read on that one index pattern and nothing else.

Paste the endpoint URL and the encoded key into the connection screen, and the first scan starts. The Elasticsearch connection guide walks through it for self-managed clusters, Elastic Cloud and OpenSearch-compatible endpoints.

What CloudThinker watches

Everything from parts one and two, on a loop instead of a calendar:

• Cluster health and allocation. Not just green, yellow or red, but why: unassigned shard counts, allocation-explain deciders, watermark distance per node.
• Shard geometry. Shard counts against data volume, median shard size, tiny-shard sprawl, hot nodes carrying more than their share.
• ILM progress. Every managed index's phase, action and step, flagging anything parked in the same step long enough to mean a stuck policy rather than a slow one, plus indices with no policy at all.
• Slow logs. Search and indexing slow logs, grouped by query shape once you ship them into an index on the cluster and grant read on it (by default they only go to log files on each node), so slow queries surface as patterns ("leading-wildcard queries against message on the biggest index") instead of ten thousand raw lines.
• Snapshot health. SLM policy success and the age of the last good snapshot, because the day you need one is the wrong day to learn it's been failing for three weeks.

The cadence is the point. A cluster that went yellow on Tuesday is a correlated finding on Tuesday, not a normalized dashboard color that someone questions during the next incident.

---

An investigation, end to end

Take the opening example from the moment the status changes. Yellow has three common readings, and each one predicts something different in the allocation explain output:

01Reading 1

Node left

Shards show NODE_LEFT with a delay. They come back on their own when the node does.

02Reading 2

Too many replicas

The same_shard decider says no everywhere. Replica count is at least the data node count.

03Reading 3

Disk watermark

The disk_threshold decider blocks one node. Something filled it, and that is the real question.

Here's the walk, call by call.

1. Confirm the symptom. _cluster/health reports "status": "yellow", "unassigned_shards": 42, "delayed_unassigned_shards": 0. No delay rules out a node that left a minute ago.

2. Ask why. POST _cluster/allocation/explain on one of the 42 returns "can_allocate": "no". The only node without a copy of that shard is es-data-4, and its disk_threshold decider says the node is above the low watermark. Reading 2 would have shown same_shard on every node; it doesn't. That's reading 3.

3. Check whether it's general growth. _cat/allocation?v&s=disk.percent:desc shows es-data-4 at 87% and its peers near 60%. Uneven disk points at one heavy index, not the whole cluster growing.

4. Find the weight. _cat/indices?v&s=store.size:desc&h=index,pri,store.size,creation.date.string puts logs-app-000001 on top at 1.4 TB with one primary. Its rollover target is 50 GB per shard.

5. Find the cause. GET logs-app-000001/_ilm/explain shows "phase": "hot", "action": "rollover", "step": "ERROR" with "failed_step": "check-rollover-ready", and step_info reads rollover_alias [logs-app] does not point to index [logs-app-000001]. GET _alias/logs-app confirms it: during last spring's migration the alias moved to a new index, but the shippers were pointed at logs-app-000001 by name. Writes kept landing on an index ILM can no longer roll over, and the error has sat in _ilm/explain ever since.

01Signal

Yellow

42 unassigned replicas in _cluster/health.

02Decider

Disk threshold

Allocation explain blocks es-data-4, past the low watermark.

03Weight

One index

logs-app-000001 at 1.4 TB, far past any rollover size.

04Cause

Stuck ILM

Rollover alias points at a different index since a migration.

05Output

Finding

The chain with evidence, and a fix waiting for approval.

The finding you receive is that chain, with the API response behind each link, plus a proposed remediation: make logs-app-000001 the write index for logs-app again with POST _aliases (is_write_index: true), POST logs-app-000001/_ilm/retry, trigger POST logs-app/_rollover, and switch the shippers back to writing through the alias. The oversized index then either ages out through the delete phase or gets reindexed into properly sized indices during a low-traffic window. Each step comes with its rollback. Same skills you built in part two, minus two hours of running them under pressure.

---

Nothing changes without your say-so

The connection is read-only, so at first CloudThinker can only observe and explain. When you want it to act, you grant scoped write access separately, and from then on it acts within Auto Mode, so you decide what runs on its own and what waits for approval. The proposed ILM policy JSON, the alias fix and the corrected shard count for the next rollover each arrive with expected impact and rollback notes.

Every finding, proposal, approval and execution lands in an audit trail: what was observed, what was proposed, who approved it, what changed and when. Escalation paths stay intact. The agent attaches the investigation; it doesn't silence the pager.

---

What a first scan typically finds

The numbers below are illustrative, a composite of a mid-market logging cluster: six data nodes, roughly 8 TB of data, a few years of organic growth. Yours will differ.

Look at the shape rather than the numbers. Nothing here is exotic. It's the hygiene list from part one, except each row arrives already correlated, with a proposed fix attached, and the table refreshes itself instead of waiting for the next audit.

---

Why this beats the traditional setup

The deeper difference is where the reasoning lives. With scripts and alerts, the cluster tells you facts and the reasoning stays in whoever is on call that night. With an agent on top, the reasoning is written down next to the evidence, so the next engineer, or the post-incident review, can check it.

---

Prompts to try in your first session

CloudThinker is conversational. You ask in plain language and get answers grounded in your cluster's actual API responses:

"Why has the cluster been yellow since Tuesday, and which index is responsible? Propose only, change nothing."

"Show me the slowest search queries from the past week, grouped by query pattern, and which indices they hit."

"Draft an ILM policy for the logs-app indices: rollover at 50 GB or 7 days, move to warm after 3 days, delete after 30. Show me which existing indices it would affect before anything changes."

Each answer cites the evidence behind it (the same _cat, _ilm/explain and slow-log data from part two), so you verify rather than trust.

What CloudThinker won't do

"AI with access to the cluster that holds your logs" deserves scrutiny. Here are the answers:

• No writes under the default connection. The API key grants monitoring privileges only. Remediation needs scoped write access you grant explicitly, and Auto Mode letting it run.
• No deleting indices, changing ILM policies or starting reindexes outside what you've allowed. You decide which of those run on their own and which wait for a named person.
• No actions outside the audit trail. Every proposal and execution is logged with evidence and approver.
• No retention decisions. It'll show you that 27 indices have no lifecycle policy and draft one. How long your compliance team needs those logs kept is a human call, and it stays one.

From cluster hygiene to standing habit

Across this series you've seen what degrades an Elasticsearch cluster, how to audit and fix each issue with native tools, and now how to make that discipline continuous. The gap between teams whose clusters stay green and teams who normalize yellow is rarely knowledge. It's cadence, and cadence is an automation problem. Teams that move from reactive firefighting to continuous, approval-gated index management typically cut search latency incidents and reclaim 20 to 40% of cluster storage within the first months, mostly from rollover and retention fixes that were always one stuck ILM step away.

Try CloudThinker free with 100 premium credits and no card required, and follow the Elasticsearch connection guide to see your own cluster's first findings within the hour.