Automating Kubernetes Cost Optimization with an AI Agent

Suppose you ran the audit and shipped a rightsizing PR last month. It's already going stale. The service whose requests you trimmed got a new feature and a new memory profile. Someone raised an HPA's minReplicas during an incident and never put it back. Two new deployments shipped this week with requests copied from a template written in 2023. Every Kubernetes cost optimization effort hits this wall eventually: the fixes don't stick, because the cluster keeps changing after you fix it.

Part one mapped the six places cluster spend leaks: overprovisioned requests, idle and underpacked nodes, orphaned volumes and load balancers, forgotten namespaces and missing autoscaling. Part two turned those leaks into a DIY audit built on kubectl top, kube-state-metrics, VPA in recommendation mode, autoscaler logs and the EKS, GKE and AKS cost views.

This part covers the step after: how CloudThinker Optimize keeps the part-two audit running against your clusters every day, how it verifies a finding before proposing anything, what a first scan tends to surface, and what it won't do until you approve it.

---

Why point-in-time rightsizing loses

A cluster with 30 active deployments and a weekly release cadence sees hundreds of changes between audits. Each one can shift the requests-versus-usage gap. A dependency upgrade changes memory behavior. A traffic shift strands a node pool. A deleted namespace leaves its PersistentVolumes and cloud load balancers behind. The leaks from part one regenerate at deploy speed, while the audit runs at calendar speed.

Native tools don't close this gap on their own. VPA recommends, but doesn't reconcile its numbers with your HPA settings and PodDisruptionBudgets. The cluster autoscaler scales nodes, but won't tell you your requests are inflated 3x, so it faithfully bin-packs the wrong numbers.

The harder gap is verification. Each finding needs checking before anyone acts: is that namespace really abandoned, does that PDB protect something real, will the smaller request survive Monday's peak. That checking is what gets skipped when the audit runs long, and skipping it is how a rightsizing PR turns into an incident.

---

Connecting: read-only RBAC, about five minutes

01Clusters and billing

Kubernetes

• Workloads, nodes, HPAs, PDBsget, list, watch only
• Usage metricsmetrics-server or Prometheus
• Cloud billingAWS, Google Cloud or Azure, read-only

Read-only ClusterRole, no secrets

02Investigation

CloudThinker

• Scan the six leaksEvery namespace and node pool, daily
• Verify each findingTraffic peaks, owners, blockers
• Propose the fixNew values, saving range, rollback

Nothing installed on your nodes

03Where the team works

Chat and Git

• Slack or TeamsFindings and approvals in the thread
• GitHub or GitLabChanges as PRs to your Helm values
• JiraFollow-ups for the owning team

Nothing new to watch

The agent connects to your cluster the way your security team would insist on anyway: a dedicated ServiceAccount bound to a read-only ClusterRole, with get, list and watch on workloads, nodes and metrics, no write verbs and no access to secrets. It needs API server reachability and nothing on your nodes. On EKS, GKE or AKS you can scope it further with your existing admission and network policies. Cloud billing context (node pricing, load balancer and volume costs) comes from your cloud connection, also read-only.

You apply a provided manifest, paste the connection details, and the first scan starts. The Kubernetes connection guide has the exact RBAC manifest, so your security review can see what's granted and what isn't.

What it scans continuously

Once connected, the agent runs the checks you ran by hand in part two, every day, across every namespace and node pool:

1. Requests against actual usage per container, over rolling 7- and 30-day windows: the P95-usage-to-request ratio that dominates most cluster bills.
2. Node utilization and bin-packing. Nodes under 30% requested, and pools where pod request shapes and instance shapes don't fit.
3. Idle and stranded nodes. Nodes kept alive by a single system pod or by a PodDisruptionBudget that blocks consolidation.
4. Orphaned resources. Released or Available PersistentVolumes, LoadBalancer Services with no healthy endpoints, unattached cloud disks left by deleted StatefulSets.
5. Autoscaling gaps. Deployments with no HPA, HPAs pinned by a stale minReplicas, pools where Karpenter or cluster autoscaler consolidation is off or blocked.
6. Non-production schedules. Dev and staging namespaces running full replica counts at 3 a.m. on a Sunday.

The checks aren't the difference. The cadence is. A deployment that ships Tuesday with 4 CPU requested and 200m used is a finding on Wednesday, not a line in next quarter's spreadsheet. Each finding carries the evidence you'd have gathered yourself: the metrics window, the current monthly cost and the projected saving as a range.

01Signal

Spend moves

Node count, requests or a cost line changes against the baseline.

02Context

Investigate

Requests, usage peaks, HPAs, autoscaler state and recent changes.

03Check

Verify safety

Peak traffic, PDBs, owners, anything the change could break.

04Output

Proposal

The change, the saving range, the rollback and the approver.

---

One finding, investigated

Here's the stale minReplicas from the opening, the way CloudThinker works it. The details are illustrative.

On the 14th, the prod-general node pool is at 18 nodes. Two weeks earlier it ran at 14, and request traffic across the cluster is flat. Three readings fit, and each predicts different evidence:

• New or padded requests. Predicts requested CPU jumping in a namespace with a new deployment or a changed request value.
• A raised replica floor. Predicts one workload with more replicas, an HPA whose minReplicas equals its current replica count, and low CPU per pod.
• Blocked scale-down. Predicts flat requests, nodes marked unneeded in the autoscaler status, and a PDB or annotation stopping eviction.

Where did the requested CPU go?

sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
-
sum(kube_pod_container_resource_requests{resource="cpu"} offset 14d) by (namespace)

Requests rose by 24 cores, all of it in the search namespace. No new deployments there. That weakens the first reading and rules out the third, since blocked scale-down wouldn't add requests. The autoscaler status ConfigMap agrees: no nodes marked unneeded.

Is it the HPA?

kubectl get hpa -n search

search-api shows MINPODS 16, MAXPODS 20, REPLICAS 16, at 9% CPU against a 70% target. The HPA would happily scale down. The floor won't let it. Each pod requests 2 CPU, so twelve extra replicas account for the 24 cores. In the repo, the Helm values change from the 2nd is titled "hotfix: raise search min replicas during reindex". The reindex job completed on the 3rd.

One more check before proposing. How many replicas does search-api actually need at peak? Over the last 14 days, its busiest hour would have needed five replicas to stay at the 70% target. A floor of six leaves headroom above that.

The finding, posted in the platform channel:

search-api HPA floor still at 16 after the reindex. minReplicas was raised from 4 to 16 on the 2nd for a reindex that finished on the 3rd. The HPA sits at 9% CPU against a 70% target. Peak demand in 14 days needed five replicas. That floor is holding about 20 cores and roughly three nodes in prod-general, around $840 a month at current on-demand pricing. Proposed: set minReplicas to 6 in the Helm values. Rollback: revert the PR. Needs approval from: the search team, which owns the chart.

The change it would open as a PR, once approved:

# charts/search-api/values-prod.yaml
autoscaling:
  enabled: true
  minReplicas: 6 # was 16, raised on the 2nd for the reindex
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70

Once the PR merges and the extra replicas are gone, the cluster autoscaler removes the empty nodes on its own. Found on day twelve instead of at the next quarterly review, with peak demand already checked.

---

You decide what the agent may touch

Scaling a staging namespace to zero on an approved schedule and changing a production node pool aren't the same kind of decision. CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Every action lands in an audit trail: what was found, what was proposed, who approved, what changed and when. Your part-two audit skills transfer directly. You review the same evidence, minus the hours spent collecting it.

---

What the first scan typically returns

These numbers are illustrative, a composite of what a first scan tends to surface across two or three mid-market clusters spending around $35K a month on compute. Your clusters will differ.

That's roughly $16,000 a month identified on day one, and the shape matches what part one predicted: requests against usage and non-prod scheduling dominate, with a long tail of orphaned resources. The value isn't exotic findings. It's that this table refreshes daily, and each row arrives with a verified, approval-gated fix instead of a ticket you still have to write.

Why this beats the traditional approach

The deeper difference is where the hours go. In the audit, most of the afternoon goes to joining tool outputs and too little to judging them. With the agent doing the joining, your time goes to the one decision that needs a person: whether the change is right for this workload.

---

Prompts to try in your first session

Optimize is conversational. Ask in plain language, and each answer cites the underlying data (the same metrics-server and kube-state-metrics evidence from part two), so you can verify it rather than trust it.

"Show me the 10 deployments with the biggest gap between CPU requests and actual P95 usage over the last 30 days. Propose only, change nothing."

"Which nodes could be consolidated right now, and what's stopping the autoscaler from doing it?"

"List every HPA whose minReplicas changed in the last 30 days, and tell me which ones are now pinned at the floor."

"What would scaling the staging namespace to zero outside business hours save us?"

What the agent won't do

"AI agent with cluster access" should make you ask. Here are the answers.

• Read-only by default. The RBAC grant has no write verbs. Remediation needs scoped write access you grant explicitly, and still goes through Auto Mode.
• No silent edits, drains or deletes. Every change shows up with its evidence, whether it ran on its own or waited for approval.
• No bypassing your guardrails. Changes go through the API server or your Git workflow like any other client, subject to your admission controllers, PDBs and quotas.
• No architectural calls. It'll tell you a PodDisruptionBudget is blocking $2,400 a month of consolidation. Whether that PDB is load-bearing is an engineering decision, and it stays yours.

If you ran the part-two audit by hand, the right mental model is that audit running daily, with a remediation queue attached. It isn't an autopilot you hand cluster-admin on day one.

From quarterly audit to control loop

You know where Kubernetes spend leaks. You know how to audit it with native tools. What's left is cadence, and that's an automation problem, not an effort problem. Teams that move from quarterly rightsizing PRs to continuous, approval-gated optimization typically cut cluster compute spend by 30 to 50% over the following months, with the biggest wins from request rightsizing and non-prod scheduling.

Related reading

• Auto Mode: decide what runs on its own
• What is continuous rightsizing?
• Optimize: continuous cloud cost optimization
• Automating AWS cost optimization with an AI agent

Try CloudThinker free with 100 premium credits and no card required, and follow the Kubernetes connection guide to see your own cluster's first findings table within the hour.