ArgoCD Automation: 7 GitOps Drift and App-Health Failures That Go Unwatched

Open the ArgoCD dashboard and count the yellow. Eleven of forty apps show OutOfSync. Nobody is alarmed. At some point the team agreed, without saying it out loud, that eleven yellow tiles is what the dashboard looks like. The day one of them matters, it will look exactly like the ten that don't.

That's the core failure of ArgoCD automation. The controller reconciles Git to cluster faithfully, but it can't tell you whether a given OutOfSync is a harmless annotation or a production hotfix about to be reverted. So the status decays. OutOfSync stops meaning "act now" and starts meaning "probably fine". Degraded scrolls past. The signal is still there. The meaning has leaked out of it.

On any fleet with more than a handful of Applications, the same seven patterns show up. This guide covers each one: what it is, why it happens, the command or status field that detects it, and what it costs you. Most checks use the argocd CLI or kubectl and take a few minutes.

This is part one of a three-part series. Part two is a hands-on guide to ArgoCD's native sync policies, health checks, and CLI triage. Part three covers automating GitOps drift triage with an AI agent.

01Status decay

Yellow nobody reads

Apps parked OutOfSync for weeks, and app-of-apps trees where a drifting child is invisible from the parent.

02Drift

Git stops being true

Hotfixes that selfHeal reverts, hotfixes that stick forever when it is off, and orphans left by chart refactors.

03Health and rollout

Red with no reason

Degraded rollups nobody drills into, and syncs that stop halfway through a wave on a failed hook.

The seven patterns in this guide, grouped by what they do to the signal.

Why the dashboard approach breaks

The usual answer to drift is "someone watches the dashboard". It fails for structural reasons, not effort.

Status is binary, intent isn't. OutOfSync says live and Git differ. It says nothing about which side is right. Answering that takes a diff, a look at who changed the field, and some memory of last night's incident.
Health rolls up, cause doesn't. One stuck Job turns the whole app Degraded. The reason sits three levels down in the resource tree.
Known noise trains people to ignore the field. An HPA-owned replicas diff that nobody adds to ignoreDifferences keeps the app yellow forever. After a month, yellow means nothing.
The knowledge lives in heads. Whether an app is "always a bit OutOfSync" is tribal. The engineer who knows has gone home.

1. Apps parked OutOfSync until the status means nothing

What it is. An Application sits OutOfSync for days or weeks. The diff is real but small: a label a mutating webhook adds, a replicas field the HPA owns, an annotation from another controller. Because nobody resolves it, it becomes background color. Then a drift that matters (a ConfigMap that no longer matches Git) lands in the same yellow pile and nobody notices.

Why it happens. Auto-sync is off for that app, deliberately or by neglect, or the diff is noise the team never taught ArgoCD to ignore. Clearing it takes judgment, so it never gets cleared.

How to detect it. List every app that isn't synced, with its health and when its last sync operation finished:

argocd app list -o json \
  | jq -r '.[] | select(.status.sync.status != "Synced")
           | [.metadata.name, .status.sync.status, .status.health.status,
              (.status.operationState.finishedAt // "never")] | @tsv'

An app whose last sync finished weeks ago and is still OutOfSync has a status that already lost its meaning.

Typical impact. No line on the bill, but a real cost: the longer apps sit yellow, the better the odds a meaningful drift goes unactioned. It's alert fatigue with a Git backend.

2. Manual kubectl hotfixes that auto-sync silently reverts

What it is. An incident at 2 a.m. Someone runs kubectl edit deployment or kubectl scale to stop the bleeding. It works. Minutes later, automated sync with selfHeal: true reconciles live state back to Git and undoes the fix. The incident reopens with no obvious cause, because the thing that broke it was a controller doing its job.

Why it happens. selfHeal reverts any live divergence from Git. It has no notion of "this divergence was an intentional emergency fix". Under pressure, people reach for kubectl, not a pull request.

How to detect it. Check whether the app self-heals, then read its recent sync activity:

argocd app get my-app -o json | jq '.spec.syncPolicy.automated'
argocd app history my-app

selfHeal: true plus a fresh automated sync right after an incident window is the fingerprint of a reverted hotfix.

Typical impact. Longer incidents and eroded trust in GitOps. The "did ArgoCD just undo my fix?" moment is what pushes teams to switch auto-sync off entirely, which throws away its main benefit.

3. Drift that sticks when selfHeal is off

What it is. The mirror image of #2. With automated sync on and selfHeal: false, ArgoCD applies new commits but does not revert manual live changes. A kubectl edit sticks indefinitely, and the app reports OutOfSync while the cluster runs config that exists nowhere in version control. Git is no longer the truth.

Why it happens. Teams turn selfHeal off to avoid #2 and forget the trade-off: every manual change now persists, and the repo drifts away from reality.

How to detect it. Diff live state against Git for a suspect app:

argocd app diff my-app

Every hunk is config running in the cluster that isn't in Git. If the app is OutOfSync and selfHeal is false, that drift lives until someone acts.

Typical impact. Silent configuration drift, the failure GitOps was meant to remove. It surfaces months later as an environment nobody can reproduce, or a disaster-recovery rebuild that comes back subtly wrong.

4. Degraded health that nobody drills into

What it is. An app's health rolls up to Degraded, and the top-level status is all anyone looks at. Underneath, one Deployment is stuck below its desired replicas, a pod is in CrashLoopBackOff, or an Ingress never got an address. The rollup is red. The reason is three clicks down, so nobody clicks.

Why it happens. Health rolls up but root cause doesn't. Degraded is common enough during deploys that people learn to wait for it to clear on its own.

How to detect it. Pull the per-resource tree and filter to the unhealthy nodes:

argocd app resources my-app
argocd app get my-app -o json \
  | jq '.status.resources[] | select(.health != null and .health.status != "Healthy")
        | {kind, name, health}'

Typical impact. Slow-burn outages: a background worker failing for a week, a canary that never went healthy, caught late because the rollup hid the one resource that mattered.

5. Sync waves and hooks that fail halfway through

What it is. ArgoCD applies resources in ordered sync waves and runs PreSync and PostSync resource hooks around them: a schema migration Job, a smoke test. When a hook fails, the sync stops partway. Some waves applied, later ones didn't. The app lands in a state that matches neither the old version nor the new one.

Why it happens. Hooks are real workloads that fail for real reasons, like a migration timeout or a flaky test. The sync halts by design, but the partial state is easy to miss if you only watch the final status.

How to detect it. Inspect the latest sync operation, including hook results:

argocd app get my-app --show-operation

Look at the operation phase and each hook's status. A Failed phase with some resources Synced and others not means you're stuck mid-wave.

Typical impact. Partial deploys: the schema migrated but the new code never rolled out, or the other way round.

6. Orphaned resources after a chart refactor

What it is. You rename a resource, restructure a Helm chart, or drop a template. ArgoCD applies the new manifests, but the old Service or ConfigMap, no longer rendered by the chart, keeps running because pruning is off. It's live, holding a name or a port, and referenced by nothing in Git.

Why it happens. Automated sync doesn't prune unless you set prune: true, because deleting things automatically feels dangerous. So refactors leave stragglers that pile up release after release.

How to detect it. Resources that exist live but not in Git are flagged in the app status:

argocd app get my-app -o json \
  | jq '.status.resources[] | select(.requiresPruning == true) | {kind, namespace, name}'

Each one is an orphan waiting for someone to decide whether it's safe to remove. Orphaned resource monitoring on the AppProject also catches resources that were never in any app.

Typical impact. Namespace clutter, stale Services still receiving traffic, and the occasional dangerous one: an old Ingress routing to a decommissioned backend.

7. App-of-apps sprawl nobody can reason about

What it is. The app-of-apps pattern (a parent Application that generates child Applications) scales well until it doesn't. Dozens or hundreds of generated apps, several layers deep, and no single view of which ones are drifting, unhealthy or stale. A child's OutOfSync at the bottom of the tree is invisible from the top.

Why it happens. The pattern hides individual apps behind a parent on purpose, which is exactly what makes fleet-wide problems hard to see. Nothing tells you "3 of your 180 child apps are Degraded".

How to detect it. Count sync and health across every app at once:

argocd app list -o json \
  | jq -r '.[] | [.metadata.name, .status.sync.status, .status.health.status] | @tsv' \
  | sort -k2

Reading this instead of the dashboard surfaces the children buried under their parents.

Typical impact. Whole apps drifting inside the tree unnoticed. The more successful your app-of-apps rollout, the bigger the blind spot.

One yellow app, worked end to end

The detection commands tell you which apps drift. They don't tell you what to do. Here's what that takes for a single app. The details are illustrative.

payments-api has been OutOfSync for nine days. Its project runs automated sync with selfHeal: false. Before touching anything, name the possible readings and what each one predicts:

01Reading 1

Manual hotfix

A person changed live state. Predicts a kubectl field manager and a timestamp inside an incident window.

02Reading 2

Controller mutation

An HPA, webhook or operator owns the field. Predicts the same diff returning after every sync.

03Reading 3

Chart bug

Git rendered something nobody meant. Predicts the diff starting at a recent commit, not a live edit.

Three readings of one OutOfSync. Each predicts different evidence, and each needs a different action.

Step 1: read the diff. argocd app diff payments-api returns one hunk: the container's resources.limits.memory is 1Gi live and 512Mi in Git. That rules out nothing yet, but memory limits aren't a field controllers usually own, so reading 2 is unlikely.

Step 2: ask who wrote the field. Server-side field ownership records it:

kubectl get deployment payments-api -n payments -o json \
  | jq '.metadata.managedFields[] | {manager, operation, time}'

One entry has a manager named kubectl-edit with a timestamp at 02:14 nine days ago. That's a person, and it lines up with the night payments-api was OOMKilled under load. Reading 1.

Step 3: check Git's side. argocd app history payments-api shows no commit touching resources since well before the incident. So this isn't a chart change. Reading 3 is out.

Conclusion. It's a hotfix that worked and was never committed. Syncing would reintroduce the OOMKill. The right action is a pull request that sets 1Gi in Git, after which the app goes green on its own. Ten minutes for one app, if you know where to look. Multiply by eleven yellow tiles.

What the status tells you, and what it doesn't

Status	What ArgoCD knows	What you still have to work out
`OutOfSync`	Live and Git differ on these fields	Which side holds the intended state
`Degraded`	Some child resource is unhealthy	Which one, and why (events, logs, exit codes)
Sync `Failed`	The operation stopped	Which wave, which hook, and what's half-applied
`requiresPruning`	This resource isn't in Git	Whether anything still depends on it
Synced and Healthy	Live matches Git	Nothing, which is the point of fixing the rest

The deeper difference is that ArgoCD answers whether, and every remaining question is why. "Why" needs evidence from three places (Git history, field ownership, Kubernetes events), and it has to be re-asked every time a new drift appears.

Why the reconciliation loop keeps generating toil

ArgoCD's reconciliation loop is continuous and tireless, which is exactly why the toil is continuous too.

01Change

Deploy or hotfix

A sync, a kubectl edit at 2 a.m., a chart refactor, a new child app.

02Residue

New drift

A partial wave, a live-only patch, an orphaned Service, a stuck rollout.

03Signal

One more yellow

The status is correct, and it looks exactly like the ten harmless ones beside it.

04Outcome

Ignored

Nobody has time to diff it, so it joins the background until it causes an incident.

The reconciliation loop runs continuously. So does the residue it leaves behind.

The seven patterns aren't a backlog you clear once. They're the steady-state output of a system doing its job. Every deploy can leave a partial sync. Every incident invites a hotfix that selfHeal reverts, or worse, doesn't. Every chart refactor sheds an orphan. Every child app widens the blind spot.

Switching off auto-sync or staring harder doesn't help. What helps is making the triage continuous: every OutOfSync diffed and classified the moment it appears, so yellow on the dashboard means something again.

Where this series goes next

You can close part of the gap with ArgoCD's own features. Part two builds it with exact configs: sync policies set deliberately, sync windows, Lua health checks for CRDs, notifications, and ignoreDifferences for known noise. The rest, the per-app judgment of whether a drift is a hotfix to keep or an accident to revert, is what part three covers. CloudThinker connects to your ArgoCD API server read-only, diffs and classifies each OutOfSync, and proposes the safe action for your approval.