Ansible AWX Integration with AI Agents: From Failed-Job Archaeology to Automation Ops

The nightly patch template fails at 02:14. What reaches you is a red icon and a link. Click it and you get 2,000 lines of output across 40 hosts, and the actual cause (a host that's been gone for weeks, an SSH key someone rotated, a task that broke after a collection upgrade) is somewhere in the middle. You find it at 09:30, after standup, if nothing more urgent shows up first.

This article is about what happens when that same failure arrives as a finding instead: which hosts, which task, what changed, and what to run next.

Part one mapped the failure and waste patterns in real AWX installations: failed jobs nobody scrolls through, inventories drifting from the cloud, snowflake extra-vars, schedules running jobs that stopped mattering, credential sprawl, unreachable hosts treated as noise. Part two built the DIY toolkit: dynamic inventories with update on launch, surveys and check mode, workflows with approval nodes, failure webhooks, and a triage script over the AWX API.

Part two ended at a ceiling. Workflows route jobs and pause for approvals, notifications say a job failed, but reading the output and deciding what it means is still manual, and it starts over with every launch. This part covers the layer that takes that on. CloudThinker sits on top of AWX, reads every job result, reconciles inventories against the cloud, and proposes approved job templates as remediation, with every launch behind the policy you set.

---

Why the red job stays red

AWX is a good controller. The reasons failed jobs pile up are structural, not a missing feature.

A failure is recorded, not explained. AWX stores the output and per-host counts for every job. It doesn't tell you that a rejected SSH key and a terminated instance both show up as dark, or that the same task now fails on every host since the last project sync.

Workflows only follow edges you drew. An on-failure edge can run a cleanup template or post to chat. It runs the same thing whatever caused the failure, because the edge was drawn before the failure existed.

The context lives outside AWX. The deploy key rotation is in the activity stream, the terminated instances are in the cloud account, the collection bump is in Git. Correlating them is a person with four tabs open.

Knowledge sits in senior heads. Whoever has run AWX for three years can glance at a recap and say "that's the bastion again." That skill doesn't scale to a few hundred jobs a week, and it goes on holiday.

---

Connecting AWX: an application token, about five minutes

01Automation controller

AWX

• Jobs and output/api/v2/jobs, stdout, job_host_summaries
• Inventories and sourcesHosts, groups, last sync state
• Templates and schedulesExtra-vars, surveys, credentials in use

Read role, OAuth token

02Investigation

CloudThinker

• Classify every red jobUnreachable, auth or task regression
• Reconcile inventoriesAgainst the live cloud account
• Propose a template runTemplate, host limit, extra-vars, rollback

Nothing installed on the controller

03Where the team works

Chat and tickets

• Slack or TeamsFindings and approvals in the thread
• JiraFollow-ups with the evidence linked
• Cloud accountRead-only, for inventory reconciliation

Nothing new to watch

CloudThinker connects the way you'd want any external system to: as a dedicated AWX user with an OAuth2 application token. Give that user the Read role on the organizations, inventories and job templates you want covered, and no Execute role at connection time. Nothing runs on your controller, no credentials are copied out, and there's no write access on day one.

You paste the AWX base URL and the token, and the first scan starts against the same endpoints you scripted in part two:

# The read-only surface CloudThinker starts from
curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/jobs/?status=failed&order_by=-finished"

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/jobs/12841/stdout/?format=txt"

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/jobs/12841/job_host_summaries/"

The AWX connection guide lists the exact role assignments so your security review has the minimal grant on record.

---

One failed job, end to end

01Signal

Job fails

A job or workflow node ends in failed. CloudThinker picks it up from the job list.

02Classify

Read the result

Host summaries first, then the first fatal line, then the last green run.

03Changes

Correlate

Credential edits, project syncs, inventory updates and cloud state in the window.

04Output

Finding

The cause, the affected hosts and templates, and a proposed next step.

Job 12841, patch-web-prod, fails at 02:14. Here's the investigation CloudThinker runs, in the order an experienced operator would.

1. Name the readings. A red patch job usually means one of three things. Hosts are gone or unroutable, which predicts dark on a subset of hosts and timeout or DNS errors. A credential broke, which predicts the same error on every host that shares it. Or a task regressed, which predicts one task failing across most hosts, starting at one project revision.

2. Check the cheapest evidence first. The host summary comes back with all 40 hosts at dark: 1 and failures: 0. Forty hosts don't all die in the same minute, so "hosts gone" is unlikely, and nothing ran far enough to regress. The first lines of output settle it:

fatal: [web-prod-01]: UNREACHABLE! => {"changed": false,
  "msg": "Failed to connect to the host via ssh: ec2-user@10.20.4.11:
  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).",
  "unreachable": true}

A rejected key, reported as unreachable. That's the auth reading.

3. Confirm with the cloud and the controller. The cloud account shows all 40 instances running and healthy, which rules out terminations. The credential attached to the template has a recent change in its activity stream:

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/credentials/31/activity_stream/?order_by=-timestamp" \
  | jq '.results[:3][] | {timestamp, operation, actor: .summary_fields.actor.username}'

The web-prod-ssh machine credential was updated at 18:05 the day before. The last successful run of patch-web-prod was at 02:14 the night before that. One more query shows six job templates use the same credential, and four of them are scheduled to run before 08:00.

4. Produce a finding, not a forward. The thread gets one message: the patch job failed on all 40 hosts because the web-prod-ssh key was replaced at 18:05 with a key the hosts don't trust. Six templates are affected and four will fail tonight. The proposal is to disable those four schedules until the key is fixed, ask the credential's owner to confirm which key is current, and once it's fixed, relaunch patch-web-prod in check mode limited to the 40 hosts before the real run. CloudThinker doesn't touch the credential secret. That's a person's call.

Compare that with what part two's classifier script would have said: "auth: check the machine credential." Correct, and a useful start. The difference is the next three questions, which credential, changed when, and what else breaks tonight, answered before anyone opens a laptop.

---

The other two readings, and inventory drift

The same loop handles the other shapes from part one.

Hosts gone or unroutable. Three hosts dark on every run with Connection timed out, while the other 37 are fine. CloudThinker checks the instance IDs against the cloud account, finds them terminated weeks ago, and proposes removing them from the inventory, or moving it to a dynamic source so it doesn't happen again.

Task regression. One task fails on 38 of 40 hosts, starting with the first job after a project sync. CloudThinker pulls the project's revision for both runs, points at the requirements.yml change that bumped a collection, and notifies the playbook's owner with the failing task and the diff. Innocent hosts stop being blamed for a playbook problem.

Inventory reconciliation runs on a timer, not only on failure. Each inventory gets compared with what actually exists in the cloud account: hosts in AWX that no longer exist, running instances that sit in no inventory, and sources whose last sync is old enough that "unreachable" really means "deleted three weeks ago."

---

Launching approved job templates as remediation

This is the step native AWX can't take on its own. When an incident elsewhere in your stack has a known fix already encoded as an AWX template (restart a stuck service, drain a node, re-run a config play to correct drift, rotate a key), CloudThinker can propose that specific template, with the extra-vars its survey requires and a host limit that covers only the affected hosts. The template stays the action layer. CloudThinker decides when it should run and against what, and runs it only inside the policy you set.

Nothing launches outside your policy

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. A re-run of an idempotent config play against a drifted staging host and a launch against production hosts are different decisions, and you choose which one needs a person.

Every finding, proposal, approval and launch lands in an audit trail: what failed, how it was classified, which template was proposed, who approved, what ran, against which hosts, with which extra-vars, and the resulting job ID.

---

What a first pass typically finds

Illustrative numbers for a mid-market setup: roughly 60 job templates, 12 inventories, a few hundred job runs a week. Yours will differ.

Almost none of it is exotic. It's the part-one list, surfaced on its own, with each row arriving as a classified finding and a proposed, approval-gated action instead of a job list somebody has to remember to open.

---

Why this beats the traditional setup

The deeper difference is where the judgment sits. Workflows and notifications are deterministic, which is what you want for routing. Reading a failure isn't deterministic: the next question depends on the last answer. That's the part CloudThinker takes on, and it's the part that used to wait until someone had time.

---

Prompts to try in your first session

You ask in plain language from chat, and every answer cites the AWX data behind it (host summaries, output, activity stream) so you can check it rather than trust it:

"The nightly patch-web-prod job failed. Tell me whether it was unreachable hosts, a credential problem or a task regression, show me which hosts and the evidence, and list any other templates that will hit the same problem tonight. Propose only, change nothing."

"Reconcile the web-prod inventory against our AWS account: which hosts no longer exist, and which running instances are missing from every inventory?"

"An incident just opened for the payments service. Is there an AWX job template that remediates it, and what host limit and extra-vars would you launch it with?"

Run the first one the next morning a nightly job goes red, and compare it with the time your team usually spends on the same output.

---

What CloudThinker won't do

"AI with access to your automation controller" should raise exactly these questions:

• Read-only by default. The token carries no Execute role. Launching a template, editing an inventory or disabling a schedule needs a write grant from you, and the launch still goes through Auto Mode.
• No launches outside the proposal. It never runs a template you haven't approved for remediation, and never outside the host limit and extra-vars in the proposal.
• No secrets handling. It can tell you a credential broke and when. It doesn't read, copy or replace the secret.
• No invisible changes. Every proposal and launch is in the audit trail with its evidence, the resulting job ID and its approver.
• No architectural decisions. It'll show you that one credential failed 40 hosts or that one collection bump broke a task. Whether to re-key the fleet or pin the collection stays an engineering call.

If you built the part-two automation, keep it. Dynamic inventories, surveys and approval nodes are the right tools for the paths you can predict. Put CloudThinker on the part you couldn't: reading the result and deciding what to run next.

From archaeology to answers

Part one showed why AWX decays when nobody reads the results. Part two showed how far native features go, and where they stop: they route and record, they don't read. Teams that move from scrolling failed output to continuous, approval-gated triage typically cut the time to answer a red job from tens of minutes to under one, and stop blaming innocent hosts for failures they never caused.

Try CloudThinker free with 100 premium credits and no card required, and follow the AWX connection guide to see your own first findings within the hour.