Vault Monitoring: The 7 Security Signals That Actually Matter

Somewhere in your infrastructure there's probably a root token that was generated during vault operator init, used to bootstrap the cluster, and never revoked. It has no TTL. It answers to no policy. It has been valid for two years, and the engineer who created it left last spring.

That token is the canonical Vault finding, and it shows why Vault monitoring is different from monitoring most infrastructure. Vault is rarely down. HashiCorp built it to stay up. What Vault does instead is drift: tokens pile up, policies widen, audit devices quietly stop being read, and the blast radius of one leaked credential grows month over month while every dashboard stays green. Uptime checks won't catch any of it.

This guide covers the seven signals that matter, from seal state to lease hygiene. For each one: what bad looks like, a real detection command, and the sober version of what happens if you ignore it. You can run every check today with the Vault CLI and a token that has read access to the sys/ backend.

This is part one of a three-part series. Part two is a hands-on Vault audit using only native tools. Part three covers running the same audit continuously with an AI agent.

01Availability

Seal and HA

A sealed standby means the failover you think you have is gone.

02Identity

Root and long-lived tokens

Setup tokens and never-expiring CI tokens that outlive their owners.

03Authorization

Wildcards and sudo

Policies widened to unblock a deploy and never narrowed again.

04Evidence

Audit devices

Missing, single, or writing to a disk that nobody reads.

05Behavior

KV access patterns

Enumeration and read spikes that look like a leaked credential.

06Lifetime

Leases and TTLs

Default maximums and endless renewal turn dynamic secrets permanent.

---

Why uptime monitoring misses Vault's real failures

Most monitoring is built around one question: is the service answering? For Vault, that question has a boring answer almost every day. The failures that hurt are about what Vault answers and to whom, and they have three structural causes.

Vault is deny-by-default, so every over-grant was written by someone. Nothing widens on its own. A wildcard path exists because an engineer typed it, usually during an incident, usually with a plan to narrow it later. There's no signal when "later" never comes.

Tokens are minted by machines and nobody owns the lifecycle. CI runners, provisioning scripts and engineers testing an auth flow all create tokens. Revoking is nobody's job, and Vault can't tell you which tokens are still needed.

The audit log is a record, not a control. Vault writes every request to its audit devices. That only helps if something reads the log. In most teams the first read happens during the incident.

---

1. Seal status and HA health

What it is. A sealed Vault serves nothing. Every application that depends on it for database credentials, API keys or PKI certs starts failing on its next renewal. In an HA cluster, a sealed or partitioned standby is quieter but worse: you think you have failover, and you don't. Nodes reseal after host restarts, storage failures or auto-unseal misconfigurations (an unreachable KMS key is a classic).

What bad looks like. Any node reporting Sealed: true, a raft cluster with fewer voters than you provisioned, or a standby that hasn't caught up with the active node.

How to detect it.

vault status
curl -s $VAULT_ADDR/v1/sys/health | jq

The sys/health endpoint is built for monitoring. It returns 200 for an unsealed active node, 429 for a healthy standby and 503 for a sealed node. Wire those status codes into whatever checker you already run. On raft clusters, add vault operator raft list-peers and alert when the peer count drops.

Typical consequence. An unnoticed sealed standby turns a routine active-node failure into a full outage of every service that fetches or renews credentials, which is commonly most of production.

2. Root tokens that never got revoked

What it is. Root tokens bypass every policy check and never expire. HashiCorp's own guidance is to revoke the initial root token as soon as setup finishes and regenerate one only for break-glass operations. In practice the setup token survives: kept "just in case", pasted in a wiki, sitting in someone's shell history.

What bad looks like. Any live token carrying the root policy outside an active, documented break-glass event.

How to detect it. Sweep every token accessor and flag the ones holding root. This needs a token with sudo on auth/token/accessors:

vault list -format=json auth/token/accessors | jq -r '.[]' | \
while read -r acc; do
  vault token lookup -format=json -accessor "$acc" | \
    jq -r 'select(.data.policies | index("root")) |
      "\(.data.accessor)  display=\(.data.display_name)  created=\(.data.creation_time)"'
done

Typical consequence. Whoever gets that token owns every secret in Vault, can disable audit logging and can rewrite every policy. The only thing between them and the KV store is the token string itself.

3. Token sprawl: long-lived and orphan tokens

What it is. Root tokens are the extreme case. The common case is hundreds of ordinary tokens minted by CI jobs, provisioning scripts and humans, with TTLs measured in months or not at all. Orphan tokens are the worst of them: they have no parent, so revoking the token tree that created them leaves them alive.

What bad looks like. Tokens with expire_time: null or multi-month TTLs attached to service accounts that only needed the token for one deploy. An accessor count that grows quarter over quarter is the sprawl signature.

How to detect it. The same accessor sweep, filtered for tokens that never expire:

vault list -format=json auth/token/accessors | jq -r '.[]' | \
while read -r acc; do
  vault token lookup -format=json -accessor "$acc" | \
    jq -r 'select(.data.expire_time == null) |
      "\(.data.accessor)  orphan=\(.data.orphan)  policies=\(.data.policies|join(","))"'
done

Typical consequence. Every long-lived token behaves like a password from 2010. Leak it once (a CI log, a laptop backup, a git commit) and it works indefinitely. On mature clusters it's common for this sweep to surface dozens of never-expiring tokens nobody can attribute.

4. Policy anti-patterns: wildcards and sudo

What it is. Vault policies are deny-by-default, which is excellent right up until someone unblocks a deploy with path "secret/*" and capabilities = ["create", "read", "update", "delete", "list"]. Wildcard paths, blanket capability lists and casually granted sudo (which unlocks protected sys/ endpoints) turn least privilege into most convenience.

What bad looks like. Policies granting * on broad path prefixes, sudo outside a small admin policy, or a widely attached policy with write access to shared mounts.

How to detect it. Grep every policy for wildcards and sudo:

for p in $(vault policy list); do
  vault policy read "$p" | grep -nE '(\*|sudo)' | sed "s/^/$p: /"
done

Then confirm what a real token can actually do on a sensitive path with vault token capabilities. Policies lie by omission. Capability checks don't.

Typical consequence. Over-broad policies mean a compromised low-value credential (a staging CI token, say) reads production secrets. The gap between what a workload needs and what its policy grants is your lateral-movement surface.

5. Missing or unread audit devices

What it is. Vault refuses to complete a request it can't audit, but only if an audit device is enabled at all. Plenty of clusters run with none, or with a file device writing to a disk nobody ships, rotates or reads.

What bad looks like. vault audit list returning nothing, a single device with no shipping pipeline behind it, or an audit log whose last human read was during the incident that prompted enabling it.

How to detect it.

vault audit list -detailed

Empty output is the finding. If a file device exists, follow up on the host: is the path on rotated storage, and does anything downstream consume it?

Typical consequence. When you investigate a suspected secret leak, the audit log is the only record of who read what and when. Without it, your incident report says "unknown scope, assume everything", which means rotating every credential in the cluster.

6. KV access patterns that point to a leaked credential

What it is. Having an audit device is step one. Reading it is the control. A leaked token doesn't announce itself. It shows up as a change in behavior: one identity suddenly reading ten times its baseline, list-then-read enumeration across a KV mount, or reads from a source address that identity has never used.

What bad looks like. A service account that normally reads three paths at boot enumerating secret/ at 2 a.m., a human identity active from two networks at once, or read volume spiking without a matching deploy.

How to detect it. Rank identities by KV read volume from the file audit device:

jq -r 'select(.type == "response" and .request.operation == "read" and
       (.request.path | startswith("secret/data/"))) |
       .auth.display_name' /var/log/vault_audit.log | sort | uniq -c | sort -rn | head

Anything at the top you can't explain deserves a per-identity look at .request.path and .request.remote_address.

Typical consequence. Catching a leaked token in hours instead of weeks is roughly the difference between rotating one credential and running a full-scope breach response. Enumeration patterns in particular mean the reader is mapping what you have. That's reconnaissance, not an integration bug.

7. Lease and TTL hygiene

What it is. Dynamic secrets are Vault's best feature, and leases are their exhaust. Every dynamic database credential and every renewable token holds a lease. When defaults are generous (the system default maximum TTL is 32 days) and consumers renew forever, the lease count climbs and credential lifetimes stretch far past anything you'd design on purpose.

What bad looks like. Lease counts growing without bound, mount-level TTLs still at defaults on sensitive mounts, and "dynamic" credentials that are effectively permanent because renewal never stops.

How to detect it. Watch the lease gauge through Vault telemetry and check what the token mount actually enforces:

curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | grep vault_expire_num_leases
vault read sys/auth/token/tune

Typical consequence. Unbounded lease growth degrades cluster performance and can destabilize storage. More quietly, long-lived dynamic credentials reopen the exact exposure window dynamic secrets were adopted to close.

---

One finding, read properly

The sweeps above produce rows. Each row still needs a judgment, and the judgment is where the time goes. Take the most common one: the never-expiring-token sweep returns an orphan token with display_name: token-ci-deploy, policies default,deploy-prod, created 14 months ago.

There are three plausible readings, and each predicts something different:

• A forgotten bootstrap token. It was created once for a pipeline that has since moved to AppRole or Kubernetes auth. If so, the audit log shows no use of that accessor for months.
• A live dependency. Some job still uses it every night. The audit log shows regular requests from one runner's address, and revoking it breaks a deploy.
• A leaked credential in use. The audit log shows requests from addresses that don't match any runner, or reads outside the paths deploy-prod was written for.

The check is the same query for all three. Vault HMACs the token in the audit log by default, but the accessor is plaintext when hmac_accessor=false is set on the device, which is worth doing for exactly this reason:

jq -r --arg acc "$ACCESSOR" 'select(.type == "request" and .auth.accessor == $acc)
  | [.time, .request.remote_address, .request.operation, .request.path] | @tsv' \
  /var/log/vault_audit.log | tail -20

No rows: revoke it. Regular rows from one runner: replace it with a short-lived auth method, then revoke. Rows from somewhere unexpected: you're in incident response, and the token gets revoked first and investigated second. Same finding, three different actions. That's why the sweep alone isn't the control.

---

Why the annual Vault review keeps missing these

None of these signals stays fixed. Tokens are minted daily, policies get widened during incidents and never narrowed, audit pipelines break on a log-shipper upgrade, and access baselines shift with every deploy.

01Review

Clean snapshot

The quarterly sweep revokes tokens and narrows policies.

02Day 2

New tokens

CI jobs and humans mint tokens with generous TTLs.

03Incident

Policy widened

A wildcard ships at 11 p.m. to unblock a release.

04Gap

Nobody looks

The audit log records it. No one queries the audit log.

A point-in-time review (quarterly, annual, pre-audit) captures a snapshot of a system that drifts continuously. Each finding is weeks old before anyone reads it, and the root token minted the day after the review stays invisible until the next one.

The deeper difference is that a review answers "what's wrong today?" while the risk in Vault is "what changed since?" Only something that watches the change stream answers the second question.

The pragmatic sequence: run the seven checks above this week, fix what they surface, then work through the DIY audit guide to script them with native tooling. Be honest about the point where scheduled scripts stop being enough.

---

Make the checks continuous

Everything in this guide can run on watch instead of on calendar. CloudThinker connects to Vault with a read-only token and watches seal and HA health, token sprawl, policy wildcards, audit-device gaps and unusual KV access continuously, raising findings the day they appear. It reads and proposes, and acts within Auto Mode, so you decide what runs on its own and what waits for approval. Part three covers how that works.

If you want to see your own cluster first, start with a question like this:

"List every Vault token with no expiry, oldest first, and tell me which ones have made no requests in the last 30 days. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with the hands-on DIY Vault audit.