How to Audit and Monitor HashiCorp Vault with Native Tools Only

Every Vault cluster ships with a one-line health check:

curl -s $VAULT_ADDR/v1/sys/health | jq

Most teams run it exactly once — the day the cluster comes up — and never wire it into anything. Then a root token from the initial setup quietly turns two years old, a policy with a wildcard path ships to prod, and the audit device that would have recorded all of it was never enabled. Vault monitoring isn't hard. It's just rarely made anyone's job.

In part one of this series we walked through the Vault health and security signals that matter — seal status, token sprawl, policy anti-patterns, missing audit devices, suspicious KV reads — and what bad looks like for each. This article is the audit itself: a full pass over a running cluster using only what HashiCorp ships — the vault CLI, the sys API, and jq. Nothing to install, nothing to buy. Budget 2–4 hours for a first pass.

Two prerequisites for everything below: VAULT_ADDR is exported, and you're authenticated with a token whose policy is broad enough for the sweeps (the token-accessor listing in pass 2 specifically requires sudo capability on auth/token/accessors).

Pass 1: Health, seal state, and HA

Start with the cluster itself, because token and policy findings are moot if the thing seals itself under you.

vault status

You want Sealed: false, and on HA clusters, a sane HA Mode (active on the node you're talking to, or standby with an active address listed). Then hit the unauthenticated endpoints — these are what your load balancer and alerting should be polling but usually aren't:

curl -s $VAULT_ADDR/v1/sys/health | jq
curl -s $VAULT_ADDR/v1/sys/seal-status | jq '{type, sealed, t, n, progress}'

sys/health encodes state in the HTTP status code, which makes it trivially alertable: 200 is initialized, unsealed, and active; 429 is an unsealed standby; 472 a DR secondary; 473 a performance standby; 501 not initialized; 503 sealed (API docs). If nothing in your stack distinguishes a 200 from a 503 on this path today, that's finding number one.

On integrated storage, also verify quorum:

vault operator raft list-peers

Three or five voters, one leader. A two-node "cluster" or a non-voter you can't explain both go in the findings doc.

Pass 2: The token sweep

Tokens are where Vault hygiene goes to die: CI jobs mint them, engineers mint them for "quick tests," and nothing ever revokes them. Vault can't list raw tokens (by design), but it can list their accessors, and each accessor resolves to full metadata (token concepts).

First, orient yourself:

vault token lookup

Then enumerate every live token in the cluster:

vault list auth/token/accessors

The count alone is informative — a mid-market cluster serving a handful of apps should be in the dozens to low hundreds. Thousands means something is minting tokens without reusing or revoking them. Now resolve each accessor and pull the fields that matter:

vault list -format=json auth/token/accessors | jq -r '.[]' |
while read -r acc; do
  vault token lookup -format=json -accessor "$acc" |
    jq -r '.data | [.display_name, (.policies|join(";")), .ttl, (.expire_time // "NEVER"), .orphan] | @tsv'
done | sort -t$'\t' -k4

Read the output for three specific problems:

• Root tokens. Any row whose policies include root. The initial root token should have been revoked after setup; if it's still here, it is your single worst finding. Once you've confirmed nothing depends on it, revoke it by accessor with vault token revoke -accessor followed by the accessor from your sweep.
• Never-expiring tokens. expire_time of NEVER on anything that isn't a deliberate, documented root-of-trust. Long-lived tokens for apps should be replaced by an auth method (AppRole, Kubernetes, cloud IAM) that issues short-lived children.
• Orphans with human names. orphan: true plus a display_name like token-jsmith usually means someone ran vault token create -orphan in 2024 and left the company in 2025.

Pass 3: Policy review — hunting wildcards and sudo

Vault token policies are deny-by-default, which means every over-grant in your cluster was written down somewhere. Read all of them:

vault policy list
vault policy read <policy-name>

For clusters with more than a handful of policies, sweep mechanically:

for p in $(vault policy list); do
  echo "== $p =="
  vault policy read "$p" | grep -nE '\*|\bsudo\b|"root"'
done

Not every hit is a problem — path "secret/data/team-a/*" is a perfectly scoped grant. What you're hunting (policy docs):

• Bare or near-bare wildcards: path "*" or path "secret/*" handed to an app that reads exactly three secrets. Scope it to the actual prefix.
• sudo capability anywhere outside a tightly held operator policy. sudo unlocks root-protected endpoints like audit device management — an app token has no business holding it.
• Full CRUD where read would do. A service that only consumes credentials should have ["read"], maybe ["read", "list"]. create/update/delete on a consumer policy means a compromised app can rewrite the secrets every other consumer trusts.

Cross-reference against pass 2: for each over-broad policy, your token table tells you exactly which live tokens hold it. That pairing — this wildcard, held by these six tokens, three of which never expire — is what turns a style complaint into a prioritized finding.

Pass 4: Audit devices and the vault audit log

Check whether the cluster is recording anything at all:

vault audit list -detailed

Empty output means every secret read since day one is unrecorded. Fix it now — it takes one command (audit docs):

vault audit enable file file_path=/var/log/vault/vault_audit.log

Two operational warnings. First, Vault blocks all requests if every enabled audit device fails to write — so enable a second device (another file path or a syslog device) and monitor disk space on both. Second, sensitive values in the log are HMAC'd by default; paths, operations, display names, and source IPs are plaintext, which is exactly what the queries below rely on.

The audit log is JSON-lines, so jq is your query engine. Which paths are hottest:

jq -r 'select(.type=="response") | .request.path' /var/log/vault/vault_audit.log |
  sort | uniq -c | sort -rn | head -20

Who has been reading one specific secret (KV v2 read paths look like secret/data/...):

jq -r 'select(.type=="response" and .request.path=="secret/data/prod/db" and .request.operation=="read")
  | [.time, .auth.display_name, .request.remote_address] | @tsv' \
  /var/log/vault/vault_audit.log

And the KV access-pattern check from part one — which identities are reading unusually many distinct secrets, the classic shape of a leaked credential being farmed:

jq -r 'select(.type=="response" and .request.operation=="read" and (.request.path | startswith("secret/data/")))
  | .auth.display_name + " " + .request.path' /var/log/vault/vault_audit.log |
  sort -u | awk '{print $1}' | uniq -c | sort -rn | head

A billing service that normally touches four paths suddenly reading forty is worth an immediate conversation, whatever the explanation turns out to be.

Pass 5: Telemetry — the numbers between audits

Vault exposes runtime metrics at sys/metrics; add a telemetry stanza with prometheus_retention_time set in the server config and you can scrape it in Prometheus format (telemetry docs):

curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/sys/metrics?format=prometheus" |
  grep -E 'vault_core_unsealed|vault_token_count|vault_expire_num_leases|vault_audit_log_request_failure'

Four gauges cover most of what this audit did by hand: vault_core_unsealed (seal state as a metric), vault_token_count and its _by_ttl / _by_policy variants (token sprawl trending up), vault_expire_num_leases (lease hygiene — steady growth means something isn't renewing or revoking cleanly), and vault_audit_log_request_failure (any nonzero value is the blocked-audit-device scenario approaching). If you already run Prometheus, wiring these in is an afternoon; alert on seal state and audit failures first.

Where the DIY audit falls short

Run this audit — a first pass on a cluster that has never had one almost always surfaces at least one live root token or wildcard policy. But be honest about the structural limits:

It's a point-in-time snapshot. Your token table and policy review describe the cluster on audit day. Tomorrow's CI run mints new tokens; next sprint's Terraform apply ships a new policy. Every finding class here regenerates continuously.

Token and policy drift lands silently between audits. Nothing in a stock Vault install pages you when a never-expiring token appears or a policy gains a wildcard. The audit log records it — but a log nobody queries is a compliance artifact, not a control. On a quarterly cadence, a bad grant lives up to ninety days before anyone looks.

It costs engineer hours, every time. The 2–4 hour pass competes with feature work, and repeat passes get deprioritized precisely when the team is busiest — which is when the sloppiest tokens get minted.

Detection isn't remediation. The sweeps produce a findings doc. Someone still has to revoke the tokens, rewrite the policies, and re-check that they stayed fixed.

What comes next

Everything above can run continuously instead of quarterly. Olivier, CloudThinker's security agent, connects to Vault with a read-only token and watches the same surfaces — seal and HA health, token sprawl, policy wildcards, audit-device gaps, anomalous KV reads — around the clock, and revokes or changes nothing without your approval. That's part three of this series.

Or see your own cluster's findings first: Try CloudThinker free — 100 premium credits, no card required.