How to Audit and Monitor HashiCorp Vault with Native Tools Only

Every Vault cluster ships with a one-line health check:

curl -s $VAULT_ADDR/v1/sys/health | jq

Most teams run it once, the day the cluster comes up, and never wire it into anything. Then a root token from the initial setup quietly turns two years old, a policy with a wildcard path ships to prod, and the audit device that would have recorded all of it was never enabled. Vault monitoring isn't hard. It's just rarely anyone's job.

Part one of this series walked through the Vault signals that matter (seal status, token sprawl, policy anti-patterns, missing audit devices, suspicious KV reads) and what bad looks like for each. This part is the audit itself: a full pass over a running cluster using only what HashiCorp ships, meaning the vault CLI, the sys API and jq. Nothing to install, nothing to buy. Budget two to four hours for a first pass.

Two prerequisites for everything below: VAULT_ADDR is exported, and you're authenticated with a token whose policy is broad enough for the sweeps. The token-accessor listing in pass 2 specifically needs sudo on auth/token/accessors.

01Pass 1

Health

Seal state, HA mode and raft quorum from the sys endpoints.

02Pass 2

Tokens

Every accessor resolved to policies, TTL and orphan status.

03Pass 3

Policies

Wildcards, sudo and write grants, paired with the tokens holding them.

04Pass 4

Audit log

Devices enabled, then jq over who read what and from where.

05Pass 5

Telemetry

Four gauges that cover the gaps between audits.

Run the passes in this order. Token findings are moot if the cluster seals under you, and policy findings only rank once you know who holds them.

Pass 1: Health, seal state and HA

Start with the cluster itself, because token and policy findings are moot if it seals under you.

vault status

You want Sealed: false, and on HA clusters a sane HA Mode: active on the node you're talking to, or standby with an active address listed. Then hit the unauthenticated endpoints. These are what your load balancer and alerting should be polling, and usually aren't:

curl -s $VAULT_ADDR/v1/sys/health | jq
curl -s $VAULT_ADDR/v1/sys/seal-status | jq '{type, sealed, t, n, progress}'

sys/health encodes state in the HTTP status code, which makes it easy to alert on: 200 is initialized, unsealed and active; 429 is an unsealed standby; 472 a DR secondary; 473 a performance standby; 501 not initialized; 503 sealed (API docs). If nothing in your stack tells a 200 from a 503 on this path today, that's finding number one.

On integrated storage, also check quorum:

vault operator raft list-peers

Three or five voters, one leader. A two-node "cluster" or a non-voter you can't explain both go in the findings doc.

Pass 2: The token sweep

Tokens are where Vault hygiene goes to die. CI jobs mint them, engineers mint them for "quick tests", and nothing revokes them. Vault can't list raw tokens (by design), but it can list their accessors, and each accessor resolves to full metadata (token concepts).

First, orient yourself:

vault token lookup

Then enumerate every live token in the cluster:

vault list auth/token/accessors

The count alone tells you something. A mid-market cluster serving a handful of apps should be in the dozens to low hundreds. Thousands means something is minting tokens without reusing or revoking them. Now resolve each accessor and pull the fields that matter:

vault list -format=json auth/token/accessors | jq -r '.[]' |
while read -r acc; do
  vault token lookup -format=json -accessor "$acc" |
    jq -r '.data | [.display_name, (.policies|join(";")), .ttl, (.expire_time // "NEVER"), .orphan] | @tsv'
done | sort -t$'\t' -k4

Read the output for three specific problems:

Root tokens. Any row whose policies include root. The initial root token should have been revoked after setup. If it's still here, it's your single worst finding. Once you've confirmed nothing depends on it, revoke it by accessor with vault token revoke -accessor and the accessor from your sweep.
Never-expiring tokens. expire_time of NEVER on anything that isn't a deliberate, documented root of trust. Long-lived app tokens should be replaced by an auth method (AppRole, Kubernetes, cloud IAM) that issues short-lived children.
Orphans with human names. orphan: true plus a display_name like token-jsmith usually means someone ran vault token create -orphan in 2024 and left the company in 2025.

Pass 3: Policy review, hunting wildcards and sudo

Vault policies are deny-by-default, so every over-grant in your cluster was written down somewhere. Read all of them:

vault policy list
vault policy read <policy-name>

For clusters with more than a handful of policies, sweep mechanically:

for p in $(vault policy list); do
  echo "== $p =="
  vault policy read "$p" | grep -nE '\*|\bsudo\b|"root"'
done

Not every hit is a problem. path "secret/data/team-a/*" is a perfectly scoped grant. What you're hunting (policy docs):

Bare or near-bare wildcards. path "*" or path "secret/*" handed to an app that reads exactly three secrets. Scope it to the actual prefix.
sudo anywhere outside a tightly held operator policy. sudo unlocks root-protected endpoints like audit device management. An app token has no business holding it.
Full CRUD where read would do. A service that only consumes credentials should have ["read"], maybe ["read", "list"]. create, update or delete on a consumer policy means a compromised app can rewrite the secrets every other consumer trusts.

Here's the narrowed version of a typical over-grant, for reference:

# Before: written during an incident to unblock a deploy
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# After: the three paths billing-api actually reads
path "secret/data/prod/billing/*" {
  capabilities = ["read"]
}

Cross-reference against pass 2. For each over-broad policy, your token table tells you exactly which live tokens hold it. That pairing (this wildcard, held by these six tokens, three of which never expire) turns a style complaint into a ranked finding.

Pass 4: Audit devices and the audit log

Check whether the cluster is recording anything at all:

vault audit list -detailed

Empty output means every secret read since day one is unrecorded. Fixing it takes one command (audit docs):

vault audit enable file file_path=/var/log/vault/vault_audit.log

Two operational warnings. First, Vault blocks all requests if every enabled audit device fails to write, so enable a second device (another file path or a syslog device) and monitor disk space on both. Second, sensitive values in the log are HMAC'd by default. Paths, operations, display names and source IPs are plaintext, which is exactly what the queries below rely on.

The audit log is JSON lines, so jq is your query engine. Which paths are hottest:

jq -r 'select(.type=="response") | .request.path' /var/log/vault/vault_audit.log |
  sort | uniq -c | sort -rn | head -20

Who has been reading one specific secret (KV v2 read paths look like secret/data/...):

jq -r 'select(.type=="response" and .request.path=="secret/data/prod/db" and .request.operation=="read")
  | [.time, .auth.display_name, .request.remote_address] | @tsv' \
  /var/log/vault/vault_audit.log

And the KV access-pattern check from part one: which identities are reading unusually many distinct secrets, the classic shape of a leaked credential being farmed:

jq -r 'select(.type=="response" and .request.operation=="read" and (.request.path | startswith("secret/data/")))
  | .auth.display_name + " " + .request.path' /var/log/vault/vault_audit.log |
  sort -u | awk '{print $1}' | uniq -c | sort -rn | head

A billing service that normally touches four paths suddenly reading forty is worth an immediate conversation, whatever the explanation turns out to be. The next section walks that conversation.

Pass 5: Telemetry, the numbers between audits

Vault exposes runtime metrics at sys/metrics. Add a telemetry stanza with prometheus_retention_time set in the server config and you can scrape it in Prometheus format (telemetry docs):

curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/sys/metrics?format=prometheus" |
  grep -E 'vault_core_unsealed|vault_token_count|vault_expire_num_leases|vault_audit_log_request_failure'

Four gauges cover most of what this audit did by hand:

vault_core_unsealed: seal state as a metric.
vault_token_count and its _by_ttl and _by_policy variants: token sprawl trending up.
vault_expire_num_leases: lease hygiene. Steady growth means something isn't renewing or revoking cleanly.
vault_audit_log_request_failure: any nonzero value means the blocked-audit-device scenario is getting close.

If you already run Prometheus, wiring these in is an afternoon. Alert on seal state and audit failures first.

One finding through the native stack

Back to the billing service. Pass 4 shows tokens from the billing-api AppRole reading 41 distinct KV paths yesterday, against a usual four. Before anyone revokes anything, there are three reasonable readings.

01Reading 1

Leaked credential

Predicts new source addresses and list-then-read enumeration.

02Reading 2

New release

Predicts the spike starting at a deploy, from known addresses.

03Reading 3

Client bug

Predicts the same few paths read over and over, no listing.

Three explanations for one read spike. Each predicts something different in the audit log.

Each one has a query that confirms or kills it.

A leaked credential shows up as source addresses the role has never used, and as list operations before the reads (an attacker maps the mount before reading it):

jq -r 'select(.type=="request" and .auth.metadata.role_name=="billing-api")
  | [.request.remote_address, .request.operation] | @tsv' \
  /var/log/vault/vault_audit.log | sort | uniq -c | sort -rn

A new release shows up as the spike starting at a deploy, with every request from the known runner and pod addresses. You need your deploy history for this, which lives outside Vault.

A client bug shows up as a handful of paths read thousands of times, usually a cache that stopped caching after an SDK upgrade. Count reads per path for the role, not distinct paths.

In this case the first query came back with two addresses that matched no pod CIDR, and 36 list operations in the four minutes before the reads. That's reading one. The response is to revoke the role's secret IDs, rotate whatever it read, and then figure out where the secret ID leaked.

Here's the honest part. Every step above was a person running jq by hand, deciding what to check next from the last answer, and pulling deploy history from another system. Nothing in the native stack would have raised the spike in the first place unless someone happened to run pass 4 that day.

What each native tool solves, and what it leaves

Native tool	What it gives you	What it still leaves to a person
`sys/health` and raft peers	An alertable seal and quorum signal	Wiring it into something that pages
Accessor sweep	Every live token with its policies and TTL	Deciding which ones are still needed
Policy review	Every wildcard and sudo grant, on paper	Knowing what each workload actually reads
Audit devices and `jq`	A complete record of who read what	Reading it, and spotting the change in behavior
Telemetry gauges	Trends for tokens, leases and audit failures	Explaining why a trend moved

01Native stack

Records

Audit devices capture every request. Telemetry counts tokens and leases.

02Your scripts

Snapshot

Sweeps and jq queries describe the cluster on the day you run them.

03Still manual

Judgment

Deciding which token is forgotten, which is live, and which is an incident.

Vault ships the evidence. Reading it on a schedule, and deciding what each row means, is left to a person.

Where the DIY audit falls short

Run this audit. A first pass on a cluster that has never had one almost always surfaces at least one live root token or wildcard policy. Then be honest about the structural limits.

It's a point-in-time snapshot. Your token table and policy review describe the cluster on audit day. Tomorrow's CI run mints new tokens, and next sprint's Terraform apply ships a new policy. Every finding class here regenerates continuously.

Drift lands silently between audits. Nothing in a stock Vault install pages you when a never-expiring token appears or a policy gains a wildcard. The audit log records it, but a log nobody queries is a compliance artifact, not a control. On a quarterly cadence, a bad grant lives up to ninety days before anyone looks.

It costs engineer hours, every time. The two-to-four-hour pass competes with feature work, and repeat passes get pushed back exactly when the team is busiest, which is when the sloppiest tokens get minted.

Detection isn't remediation. The sweeps produce a findings doc. Someone still has to revoke the tokens, rewrite the policies and check that they stayed fixed.

The approval question you'll meet again

Every fix in this post is a write against a secrets store, and you already made one decision about it: which changes you'd run without a second pair of eyes. In part three, CloudThinker acts within Auto Mode, so you draw that same line: what runs on its own and what waits for approval. For a secrets store, most teams keep revocations and policy changes waiting for a named person a long time, and that's fine.

What comes next

Everything above can run continuously instead of quarterly. CloudThinker connects to Vault with a read-only token and watches the same surfaces (seal and HA health, token sprawl, policy wildcards, audit-device gaps, unusual KV reads) around the clock. It runs the read-spike investigation above on its own, and revokes or changes nothing without your approval. That's part three of this series.

To try it against your own cluster first, ask something like:

"Baseline KV reads per AppRole over the last 30 days and flag any role reading more than three times its usual number of distinct paths. For each one, check source addresses and list operations. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required.