A DIY Kubernetes Cost Optimization Audit with Native Tools

Run this on your most expensive cluster before reading further:

kubectl describe nodes | grep -A 8 "Allocated resources"

Then run kubectl top nodes next to it. If most nodes show 70% or more of CPU requested and 15% or less actually used, you're looking at the most common Kubernetes cost problem there is. The nodes can't be scaled away, because the scheduler thinks they're full. They're mostly air.

Block out one afternoon and you can walk out with a findings document that names every over-requested deployment, every half-empty node and every workload blocking scale-down, using only tooling that ships with Kubernetes or comes free from your cloud provider: kubectl and metrics-server, kube-state-metrics, the Vertical Pod Autoscaler in recommendation mode, cluster-autoscaler logs, and the cost views in the EKS, GKE and AKS consoles.

Part one mapped where cluster spend leaks: overprovisioned requests and limits, idle and underpacked nodes, orphaned volumes and load balancers, forgotten namespaces and missing autoscaling. This part is the audit itself, grouped by tool, because each tool answers a different question.

01Tool 1

kubectl top

Requests against live usage, pod and node, right now.

02Tool 2

kube-state-metrics

The same gap per namespace, over a full week.

03Tool 3

VPA, recommend only

What each request should be, from observed traffic.

04Tool 4

Autoscaler logs

Nodes it wants to remove and what blocks it.

05Tool 5

Cost consoles

Dollars per namespace, so findings get prioritized.

Each tool answers one question. Run them in this order and each narrows the next. Budget 3 to 4 hours per cluster.

Tool 1: kubectl top and metrics-server, the requests-versus-usage gap

The biggest Kubernetes waste source is the gap between what pods request and what they use. The scheduler and the cluster autoscaler both work from requests, not usage, so this gap is priced into every node you run.

First, confirm metrics-server is installed (EKS needs you to install it; GKE and AKS ship it by default):

kubectl get deployment metrics-server -n kube-system

Then look at the node picture:

kubectl top nodes

How to read it. kubectl top nodes shows actual CPU and memory usage per node. Now compare it with what's reserved:

kubectl describe nodes | grep -A 8 "Allocated resources"

For each node you get two very different numbers: allocated (the sum of pod requests, what the scheduler thinks is used) and live usage from top. A node at 78% CPU requested and 11% CPU used is the classic signature. On a first audit of a mid-size cluster, cluster-wide CPU usage under 30% of requests is common. Well-tuned clusters run at 50 to 70%.

Find the worst offenders at the pod level:

kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

And dump every workload's resource requests to compare:

kubectl get pods -A -o custom-columns=\
'NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'

Cross-reference the two lists. A pod requesting 2 CPU while top shows 40m is a 50x overask. Also note every pod whose request columns come back empty. Unset requests are the opposite failure (they schedule greedily and get evicted unpredictably), and they'll skew every number in the rest of this audit.

One caveat before you file tickets: kubectl top is a point-in-time sample. A batch job that spikes nightly will look idle at 2 p.m. The next two tools fix that.

Tool 2: kube-state-metrics and Prometheus, requests over time

kube-state-metrics exports the declared state of the cluster, including every container's requests and limits, as Prometheus metrics. If you run any Prometheus-compatible stack (kube-prometheus-stack installs both), you can turn the spot check above into a seven-day view.

Requested CPU per namespace:

sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

CPU actually used per namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)

The number that drives the whole audit is utilization of requests, and you want its peak over a full week, not its average:

max_over_time(
  (
    sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
    /
    sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
  )[7d:5m]
)

Repeat with resource="memory" and container_memory_working_set_bytes (no rate()) for the memory side.

How to read it. A namespace whose peak never goes above 25% of requested CPU across a full week, busiest hour included, is carrying at least two to three times more request than it needs. Then rank by absolute waste, not by ratio. A 40-core namespace at 30% is worth more than a 2-core namespace at 5%:

topk(10,
  sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
  -
  sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
)

That list, in idle cores, is your remediation priority. It usually surfaces the same three or four services that dominate the bill.

Tool 3: VPA in recommendation mode, the right-size numbers

You now know which workloads are over-requested. The Vertical Pod Autoscaler tells you what their requests should be, and with updateMode: "Off" it only recommends. It never touches a running pod. GKE has VPA built in (gcloud container clusters update CLUSTER --enable-vertical-pod-autoscaling, see the GKE docs). AKS offers it as a managed add-on. On EKS you install the upstream version:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Create one VPA object per suspect workload from your Tool 2 ranking:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api-recommender
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: 'Off'

Give it a day or two of real traffic, then read the recommendation:

kubectl describe vpa checkout-api-recommender -n prod

How to read it. The Status.Recommendation block gives you four numbers per container. Target is the request VPA would set: your right-size figure. Lower Bound is the floor below which the pod is likely to be throttled or OOM-killed. Upper Bound is the ceiling worth considering for spiky workloads. Uncapped Target is what VPA wanted before any policy constraints. A deployment requesting 2000m CPU with a target of 312m is your finding, with the safe new number attached.

Two cautions. Let VPA watch at least one full weekly traffic cycle before trusting a target for anything customer-facing. And don't let VPA in Auto mode and an HPA both act on CPU for the same deployment; they'll fight. Keep limits at 1.5 to 2x the new request rather than removing them.

Tool 4: cluster-autoscaler logs, idle nodes and scale-down blockers

Over-requests waste money inside nodes. The autoscaler tells you about waste at the node level: nodes it knows are unneeded but can't remove. On EKS with the self-managed cluster autoscaler, start with the status ConfigMap:

kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

Then search the logs for scale-down decisions:

kubectl -n kube-system logs deploy/cluster-autoscaler --tail=2000 \
  | grep -iE "scale.?down|unneeded|no.?scale"

On GKE and AKS the autoscaler is managed and its pod isn't visible. On GKE, read the cluster autoscaler visibility events in Cloud Logging (the cluster-autoscaler-visibility log for the cluster). On AKS, use kubectl get events -A --field-selector source=cluster-autoscaler.

How to read it. Two findings live here. First, nodes marked unneeded for long stretches that never terminate: each one is a full instance of pure waste. Second, and more useful, the reasons scale-down is blocked, which the logs state outright: pods without a controller, pods using local storage, restrictive PodDisruptionBudgets, or the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation. List the PDBs that allow zero evictions, since one of them can pin a whole node:

kubectl get pdb -A -o json \
  | jq -r '.items[] | select(.status.disruptionsAllowed == 0)
    | "\(.metadata.namespace)/\(.metadata.name)\tminAvailable=\(.spec.minAvailable // "-")\tmaxUnavailable=\(.spec.maxUnavailable // "-")"'

Every blocker you fix compounds. It doesn't just remove one node today, it lets the autoscaler keep packing nodes every day after.

While you're at the node level, check bin-packing directly. If kubectl describe nodes shows most nodes at 50 to 65% requested, your pods' request shapes don't fit your instance shape. A different node size, or a second node group for large pods, often recovers 10 to 20% of node spend with no workload changes.

Tool 5: cloud cost views, dollars instead of cores

Everything so far is in cores and GiB. Your cloud's billing console turns it into dollars per namespace or workload, which is what turns an audit into an approved ticket.

EKS: turn on split cost allocation data for EKS under Billing and Cost Management → Cost Management Preferences. After about 24 hours, Cost Explorer can group EKS spend by cluster, namespace and workload.
GKE: turn on GKE cost allocation (gcloud container clusters update CLUSTER_NAME --enable-cost-allocation), then group billing reports by namespace or cluster label. The GKE console's Clusters → Cost optimization tab also charts requested against used per cluster, with no setup.
AKS: turn on the AKS cost analysis add-on (az aks update -g RG -n CLUSTER --enable-cost-analysis, Standard or Premium tier), then open Cost Management → Cost analysis and pick the Kubernetes views for namespace and asset breakdowns.

How to read it. Put each namespace's monthly dollar figure next to its peak utilization from Tool 2. "Namespace X costs $4,200 a month and peaks at 22% of what it requests" is the sentence that gets a right-sizing ticket prioritized.

What each tool answers

Tool	Answers	Doesn't answer
`kubectl top`	Usage against requests, right now	Whether right now is typical
kube-state-metrics + PromQL	Peak utilization per namespace over a week	What the new request should be
VPA (`updateMode: "Off"`)	The right-size request per container	Which node it frees, or what that saves
Autoscaler logs and PDBs	Which nodes are unneeded and why they stay	Whether a blocking PDB is load-bearing
Cloud cost views	Dollars per namespace and workload	Which change to make first

Read the right-hand column top to bottom and you have the honest ceiling of the native toolchain: every tool measures, none of them joins the measurements or decides.

Where the DIY audit stops

Run this audit. It works. But know its limits before you make it your strategy.

01Cadence

A snapshot

Requests drift with the next deploy. VPA targets describe this week.

02Cost

Recurring hours

One cluster is an afternoon. Ten clusters on three clouds is headcount.

03Correlation

Joined by hand

Nothing links a VPA target to the node it would free or the dollars it saves.

04Action

PRs for others

Every finding becomes a change to someone else’s Helm values.

The native toolchain is good at measurement. It stops where verifying and acting begin.

It's a snapshot. Requests drift the day after you tune them, because every deploy can reset them. The VPA targets you collected this week describe this week's traffic.

It's engineer hours, recurring. One cluster, one afternoon. Ten clusters on three clouds, every quarter, is real headcount, and it's the first thing dropped in a busy sprint.

The tools don't talk to each other. You joined top output, PromQL, VPA objects, autoscaler logs and a billing console by hand. Nothing connects a VPA target to the node it would free and the dollars that node costs.

Nothing here remediates. Every finding becomes a PR to someone else's Helm values, and tickets without an owner are where the savings quietly expire.

The work left after the native tools is verification and judgment: is that idle namespace really abandoned, does that PDB protect something real, will the new request survive Monday's peak. Part three hands that work to an agent that runs this audit every day.