ArgoCD Automation With Native Tools: Sync Policies, Health Checks, and Safe Reconciliation

Two booleans in an Application spec decide whether the ArgoCD controller is a safety net or a chainsaw: prune and selfHeal. Get them right and drift heals itself while orphaned resources disappear cleanly. Get them wrong and a teammate's emergency kubectl edit vanishes ten seconds later, or a resource dropped from the chart survives forever. This guide treats those flags, and the rest of ArgoCD's native automation, with the caution they deserve.

Part one walked the failure patterns in ArgoCD-run GitOps: apps parked OutOfSync until the status lost meaning, hotfixes silently reverted (or silently kept), Degraded health nobody drills into, sync waves failing halfway, and app-of-apps sprawl. This part is the hands-on tuning pass, using only ArgoCD's own features and the argocd CLI. It ends on the one question none of it can answer.

Everything here follows the official Argo CD documentation. Test against a non-production app first, since several settings change reconciliation behavior across the fleet.

01Policy

Sync policy

Decide per app whether ArgoCD reconciles, prunes and self-heals on its own.

02Timing

Sync windows

Constrain when automated syncs may run, with a manual override for on-call.

03Truth

Health and diffing

Lua health checks for CRDs and ignoreDifferences for fields controllers own.

04Signal

Notifications

Push sync failures and Degraded health to Slack or a webhook.

05Response

CLI triage

get, diff, history and rollback when a human has to look.

The native toolchain in the order this guide configures it.

Sync policies: automated or manual, chosen deliberately

An Application either reconciles on its own or waits for a person. That choice, and the two flags attached to it, is the highest-leverage decision in your setup.

Manual sync, the conservative default, leaves the app OutOfSync until someone syncs it. Automated sync reconciles whenever Git and live state diverge:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

Now the two flags and their blast radius (automated sync docs).

prune: true lets ArgoCD delete resources that no longer exist in Git. Without it, removing a Deployment from your manifests leaves the live one orphaned. With it, a bad chart refactor that drops a resource by accident deletes it from the cluster on the next sync. Prune is right for most apps. Pair it with PruneLast=true, so deletions happen after everything else is healthy, and with review discipline on pull requests that delete manifests.

selfHeal: true reverts any live change that diverges from Git, continuously. This is what undoes a manual kubectl scale during an incident. Turning it on declares "Git is the only way to change this app". That's the point of GitOps, but only once your team actually ships hotfixes through Git. Turn it on too early and you train people to fight the controller.

A sane rollout: start automated with prune: true and selfHeal: false. Watch what drifts for a few weeks. Once manual changes have moved into Git, enable selfHeal.

Sync windows: when reconciliation is allowed

Auto-sync at 2 a.m. on a Saturday is how a merged-but-not-ready pull request turns into a weekend page. Sync windows constrain when automated syncs run, scoped by application, namespace or cluster, and live on the AppProject (sync windows docs):

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  syncWindows:
    - kind: deny
      schedule: '0 0 * * SAT'
      duration: 48h
      applications:
        - 'payments-*'
      manualSync: true

This deny window starts at midnight Saturday and blocks automated syncs for 48 hours, while manualSync: true still lets on-call sync deliberately. Check what's active with:

argocd proj windows list production

Health checks, including custom Lua for CRDs

ArgoCD ships health assessments for built-in kinds. A Deployment is Healthy when its rollout completes, Progressing while it rolls, and Degraded when it exceeds its progress deadline. The gap is your CRDs. A custom resource with no health logic reports Healthy the moment it exists, even if its controller never reconciled it. That false green hides real breakage.

Close the gap with a Lua health check in argocd-cm (resource health docs):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.example.com_Database: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for i, condition in ipairs(obj.status.conditions) do
        if condition.type == "Ready" and condition.status == "True" then
          hs.status = "Healthy"
          hs.message = condition.message
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for Database to become Ready"
    return hs

Now the CRD only reports Healthy once its own controller sets a Ready condition, and the parent app's health rollup becomes something you can trust.

Resource hooks and sync waves: ordering the rollout

Half-applied syncs usually trace back to ordering. Sync waves apply resources in ascending order, and a wave doesn't start until the previous wave's resources are Healthy. Hooks run Jobs at defined phases: PreSync for migrations, PostSync for smoke tests, SyncFail for cleanup (hooks docs, sync waves docs):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: '-1'
spec:
  backoffLimit: 1
  template:
    spec:
      containers:
        - name: migrate
          image: registry.example.com/payments-migrate:1.42.0
      restartPolicy: Never

The migration runs before the app's Deployments roll, and must succeed first. If it fails, the sync stops there. That's the "failing halfway" symptom from part one, working as designed. HookSucceeded deletes completed hooks so they don't linger, and a pinned image tag keeps the migration that ran traceable to a commit.

Notifications: triggers and templates to Slack or webhooks

Toil grows when a Degraded app waits for someone to notice it. Argo CD Notifications pushes state changes to Slack, webhooks or email through triggers (when to fire) and templates (what to say), configured in argocd-notifications-cm (notifications docs):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-degraded]
  template.app-degraded: |
    message: |
      Application {{.app.metadata.name}} is Degraded.
      Sync: {{.app.status.sync.status}}

Subscribe an app with an annotation:

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-health-degraded.slack: platform-alerts

Fire on on-sync-failed and on-health-degraded at minimum. Skip on-sync-running, which is noise.

ignoreDifferences: silencing known-noisy fields

Some fields are written by the cluster, not by you: an HPA rewriting replicas, an admission controller injecting fields, cert-manager rotating a secret. Left alone, they keep an app permanently OutOfSync and teach everyone to ignore the status. ignoreDifferences tells ArgoCD to stop diffing specific paths (diffing docs):

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: payments-api
      jsonPointers:
        - /spec/replicas
    - group: '*'
      kind: '*'
      managedFieldsManagers:
        - kube-controller-manager

Ignoring /spec/replicas on an HPA-managed Deployment is correct. Ignoring it everywhere would hide the manual scale you do want to catch. Scope every rule as narrowly as the noise requires and no wider. If automated sync should also leave those fields alone when it applies, add the RespectIgnoreDifferences=true sync option.

CLI triage, worked through one drifted app

When an app is OutOfSync or Degraded, four commands tell you what happened and let you act (CLI reference). Here they are on one case. The details are illustrative.

The Slack notification says checkout-web went Degraded and OutOfSync twenty minutes after a deploy. Three readings fit: the new revision is broken, someone changed live state during the rollout, or a field the cluster owns is showing up as drift and the Degraded is unrelated.

Start with the summary. Sync status, health and per-resource state in one view:

argocd app get checkout-web

The Deployment is Degraded and the rest is Healthy. So whatever happened is in one workload.

Then see the drift. app diff compares live state against the target revision, field by field:

argocd app diff checkout-web

The only hunk is spec.replicas: 4 live, 2 in Git. The Deployment has an HPA. That's the cluster owning the field, which explains the OutOfSync and says nothing about the Degraded. Reading three explains the yellow. It doesn't explain the red.

Read the history. The Degraded started twenty minutes after a deploy, so look at what shipped:

argocd app history checkout-web

The latest revision is the deploy. kubectl describe deployment checkout-web shows new pods failing readiness with ProgressDeadlineExceeded on the rollout. The new revision is broken. Reading one.

Roll back. Rollback is refused while automated sync is enabled, because the controller would immediately sync forward again. Disable it first, then roll back to the last good history ID:

argocd app set checkout-web --sync-policy none
argocd app rollback checkout-web 42

Then fix forward in Git, add the HPA ignoreDifferences rule so the next incident isn't muddied by a fake drift, and re-enable automated sync. Ten minutes for someone who knows ArgoCD well. Most of that time was ruling out a reading that turned out to be noise.

What native automation covers, and what it doesn't

Problem from part one	Native feature	What's left for a person
Stale `OutOfSync` from known noise	`ignoreDifferences`, `RespectIgnoreDifferences`	Spotting the noise and writing the narrow rule
Hotfixes reverted or kept	`selfHeal` on or off	Deciding, per drift, which was right
`Degraded` nobody opens	Lua health checks, notifications	Drilling into events, logs and exit codes
Partial syncs	Waves, hooks, `SyncFail`	Working out what's half-applied and what to do
Orphans after refactors	`prune: true`, `PruneLast`	Confirming nothing still depends on the orphan
Risky timing	Sync windows	Choosing when an exception is worth it

The deeper difference: every native feature is a rule decided in advance. Rules are excellent for the cases you predicted. The right column is all the cases you didn't.

The honest part: where native automation stops

You should configure all of this. It turns a fleet of ambiguous OutOfSync apps into one where each remaining diff means something. But native ArgoCD automation has a hard ceiling.

Auto-sync reconciles state. It can't judge intent. When app diff shows a live value that doesn't match Git, ArgoCD knows that they differ, not why. It can't tell whether the drift was a deliberate hotfix worth committing back, or an accidental kubectl edit against the wrong context that should be reverted now.

01selfHeal: true

Reverts everything

The accidental edit and the 2 a.m. fix that kept production up both go back to Git.

02selfHeal: false

Keeps everything

Both changes stick, and Git quietly stops describing the cluster.

03Missing option

Judge the drift

Read the diff, find who changed it and why, then keep, revert or escalate.

Native automation offers the first two. The third is still a person at a terminal.

Health is a status, not a diagnosis. A Lua check tells you a CRD is Degraded. It doesn't pull the resource's Kubernetes events, read the failing Job's logs, or tell you the migration failed because a column already exists.

Notifications move toil. They don't remove it. A Slack message about a Degraded app is a page, not a resolution. Someone still runs argocd app get, reads the diff, decides and acts.

So the native setup is real and worth the effort. It's the difference between a fleet that pages meaningfully and one that cries wolf. But it stops exactly where judgment starts: is this drift a hotfix to keep or an accident to revert? Every OutOfSync is still a manual investigation.

What comes next

That investigation (diff Git against live, find who changed the field, check the Degraded resource's real state, propose the safe action) is what part three automates. CloudThinker connects read-only to your ArgoCD API server, watches sync and health across the fleet, triages each OutOfSync, and proposes keep-and-commit, revert or escalate. It acts within Auto Mode, so you decide what runs on its own and what waits for approval.