Ansible AWX Integration: Automation Ops with AWX's Native Tools

AWX workflows have a node type that quietly admits where the platform stops: the approval node. Drop it into a workflow and the run pauses until a person clicks approve or deny. It exists because some steps shouldn't fire without someone looking first, and that's good engineering. It's also where the DIY story ends. The person clicking approve still had to open last night's failed job, read the output, decide whether the failure was a bad host or a bad play, and carry that context in their head. AWX routes the decision to a human. It doesn't make the decision any easier.

Part one mapped the failure and waste patterns: failed jobs nobody reads, inventories drifting from the cloud, snowflake extra-vars, schedules running jobs that stopped mattering. This article is the hands-on Ansible AWX integration work: tightening each of those with AWX's own features. Every command is runnable with the awx CLI (from the awxkit package) or plain curl, and the last section is honest about what these tools still can't do.

We'll go feature by feature, in the order a job meets them:

01Inventory

Sync on launch

A dynamic source refreshes hosts from the cloud before every job.

02Template

Survey and check

Typed inputs instead of free-form vars, and a dry run before risky changes.

03Workflow

Converge, approve

Run only when every branch passed, and pause for a person before the risky node.

04Signal

Notify on failure

A webhook with the job ID lands where an owner already looks.

05Audit

API scripts

Host summaries, stdout and schedules, pulled and checked on a timer.

---

Before you start: point the CLI at your instance

The awx CLI is a thin wrapper over the REST API at /api/v2/, so anything below works either way. Install it and export a token:

pip install awxkit

export TOWER_HOST=https://awx.example.com
export TOWER_OAUTH_TOKEN=$AWX_TOKEN

awx me -f human   # confirms who the token belongs to

Use a dedicated service user for scripts, not your own account, so the audit log shows what was automation and what was a person.

Feature 1: Dynamic inventories with update on launch

A static inventory rots the moment someone launches an instance. The fix is a dynamic inventory: an inventory source backed by a plugin (Amazon EC2, Azure, Google Compute Engine, OpenStack, VMware) that syncs hosts from the source of truth.

Console path: Resources → Inventories → Add inventory, then Sources → Add with your cloud credential and source type. The setting that matters most is Update on launch. Turn it on and every job that targets the inventory syncs it first, with a Cache timeout so back-to-back jobs don't hammer the cloud API (AWX inventories docs).

A source config for EC2 that groups hosts by tag and only keeps running instances:

# Source variables on an Amazon EC2 inventory source
regions:
  - ap-southeast-2
  - us-east-1
filters:
  instance-state-name: running
keyed_groups:
  - key: tags.Role
    prefix: role
hostnames:
  - private-ip-address

Trigger a sync and check the result from the CLI:

awx inventory_sources update 42 --monitor
awx hosts list --inventory 7 -f human --filter 'name,enabled'

How to read it. After a sync, compare AWX's host count with the cloud's. A gap means either stale hosts (they'll show up as unreachable) or instances AWX never learned about. Use smart inventories with a host_filter, or constructed inventories on recent AWX versions, to slice synced hosts into groups by tag instead of maintaining membership by hand.

Feature 2: Job template design: surveys, extra-vars discipline, check mode

A job template is where operational discipline lives or dies. Three habits keep templates from becoming snowflakes.

Surveys instead of free-form extra-vars. A survey (Templates → (template) → Survey) turns "whatever the operator remembered to type" into a typed form with defaults, choices and required fields. It documents itself, so the next person doesn't have to guess what target_version means.

Extra-vars discipline. Keep template-level extra_vars minimal, and put defaults in the playbook's group_vars or role defaults in Git. Enable Prompt on launch only for the one or two values that really vary per run. Every promptable field is a place a bad value can enter unseen.

Check mode as a dry-run gate. Ansible's check mode runs plays without changing anything and reports what would change. In AWX, set the template's Job type to Check, or enable Prompt on launch for job type so you can choose per run:

awx job_templates launch 15 \
  --job_type check \
  --extra_vars '{"target_version": "2.4.1"}' \
  --monitor

How to read it. Read the changed-task count the way you'd read a terraform plan. A clean check run that reports zero changes on hosts you expected to change is its own warning: the play probably isn't matching the state you think it is (Ansible check mode docs).

Feature 3: Workflows: convergence and approval nodes

Workflows chain templates into a graph with on-success, on-failure and always edges. Two node types do the heavy lifting.

Convergence. When several parent branches feed one node, set the node's convergence to All so it runs only after every parent succeeds. A "deploy" node behind "build" and "smoke-test" should never run on a partial pass.

Approval nodes. In the Workflow Visualizer, add a node and choose Approval. The run pauses until someone with approve rights decides. Set a timeout so an unanswered approval fails instead of hanging forever (AWX workflows docs).

Launch a workflow and follow the whole graph:

awx workflow_job_templates launch 9 --monitor

Pending approvals are queryable too, which is handy for a chat reminder:

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/workflow_approvals/?status=pending" \
  | jq -r '.results[] | "\(.id)\t\(.name)\t\(.created)"'

How to read it. Put the risky node behind an approval or a check-mode predecessor, every time. Convergence is the correctness guard; approval is the judgment guard. Neither one tells the approver whether last night's failure was transient.

Feature 4: Notifications on failed jobs

Silent failures were the core problem in part one. Wire failures to a place someone owns.

In the UI: Administration → Notifications → Add, pick a type (Slack, webhook, email, PagerDuty and others), then open a template's Notifications tab and toggle Failure for it. Through the API, create a webhook notification template and attach it to a job template's error list (AWX notifications docs):

# 1. Create the webhook notification template
curl -s -X POST https://awx.example.com/api/v2/notification_templates/ \
  -H "Authorization: Bearer $AWX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "failed-jobs-webhook",
    "organization": 1,
    "notification_type": "webhook",
    "notification_configuration": {
      "url": "https://hooks.example.com/awx",
      "http_method": "POST",
      "headers": {}
    }
  }'

# 2. Attach it to template 15 so it fires on failure
curl -s -X POST https://awx.example.com/api/v2/job_templates/15/notification_templates_error/ \
  -H "Authorization: Bearer $AWX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"id": <notification_template_id>}'

How to read it. Failure is the trigger that matters. Success notifications train people to mute the channel. Make sure the payload carries the job ID so the responder lands on the output instead of hunting for it.

Feature 5: The API for triage: stdout and host summaries

Everything the UI does is a REST endpoint. This is where you script the reading from part one.

Launch a template and keep the job ID:

job_id=$(awx job_templates launch 15 --extra_vars '{"app": "api"}' -f jq --filter '.id')

Pull the plain-text output, or just the lines that matter:

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/jobs/${job_id}/stdout/?format=txt" \
  | grep -n -E "^fatal:|UNREACHABLE!|PLAY RECAP" -A 3

Pull the per-host summary to see what happened beyond pass or fail:

curl -s -H "Authorization: Bearer $AWX_TOKEN" \
  "https://awx.example.com/api/v2/jobs/${job_id}/job_host_summaries/" \
  | jq '.results[] | {host: .host_name, ok, changed, failures, dark}'

Put the two together and you get a first-pass classifier for any failed job. It only looks at counts, but it sorts most red jobs before anyone opens the output:

#!/usr/bin/env bash
# classify-job.sh <job_id>: rough first read of a failed AWX job
set -euo pipefail
api="https://awx.example.com/api/v2"
auth="Authorization: Bearer $AWX_TOKEN"

summ=$(curl -s -H "$auth" "$api/jobs/$1/job_host_summaries/?page_size=200")
total=$(jq '.results | length' <<<"$summ")
dark=$(jq '[.results[] | select(.dark > 0)] | length' <<<"$summ")
failed=$(jq '[.results[] | select(.failures > 0)] | length' <<<"$summ")

if curl -s -H "$auth" "$api/jobs/$1/stdout/?format=txt" \
     | grep -q -E "Permission denied|Missing sudo password|Incorrect sudo password"; then
  echo "auth: check the machine credential and become settings"
elif (( dark > 0 && failed == 0 )); then
  echo "unreachable: $dark of $total hosts, check inventory and network"
elif (( failed * 2 > total )); then
  echo "likely regression: $failed of $total hosts failed, check the first fatal task and project revision"
else
  echo "mixed: $failed failed, $dark unreachable, read the output"
fi

How to read it. dark is the unreachable count: hosts AWX couldn't connect to. failures is tasks that ran and errored. High dark with zero failures is almost never the playbook. It's inventory, network, or an SSH key that stopped working (a rejected key also counts as unreachable), which is why the script checks the output for Permission denied first. The script gets you that far. The next question (is it the deploy key that rotated Tuesday, or the collection bump in the last project sync?) is still yours.

Feature 6: Schedule hygiene

Schedules are where dead automation hides. AWX schedules use iCal RRULE recurrence, and nobody audits them.

awx schedules list --all -f human \
  --filter 'id,name,enabled,next_run,unified_job_template'

How to read it. For each enabled schedule, check the template's last few jobs. A schedule firing a nightly play against an environment retired two quarters ago is pure noise, and it manufactures the failures that teach people to ignore the dashboard. Disable before deleting so the definition survives while you confirm nobody needs it:

awx schedules modify <schedule_id> --enabled false

---

What native AWX covers, and what it leaves to you

The deeper difference is between structure and judgment. Every native feature above adds structure: it makes the right thing easier and the wrong thing visible. None of them reads a red job and decides what it means. That part still costs a person at a desk with four tabs open.

The honest part: where DIY automation ops stops

Run all of this. Dynamic inventories, survey-driven templates, check-mode gates, convergence and approval nodes, failure webhooks, scripted triage and schedule audits are the difference between AWX as a mess and AWX as a platform. But be clear about the ceiling.

01Workflows

Route

Run the next node on success or failure, and pause at an approval node.

02Notifications

Deliver

Tell an owner a job failed, with a link to the output.

03Still manual

Read and decide

Classify the failure, find the cause, pick the template to run and the hosts to run it on.

Structure routes decisions; it doesn't make them. An approval node waits for a person. A convergence node enforces "all branches passed." Neither reads the failed output or decides whether a dark host is a fluke or an outage.

The reading is still manual. The classifier script tells you five hosts were unreachable. Whether that's a network blip, an expired credential or dead infrastructure, and what to run next, is still you correlating across the cloud console, the Git log and the job history.

It's point in time. Every audit above is a snapshot. The inventory drifts again next sprint, a new snowflake template lands, someone adds a schedule nobody will revisit.

Detection isn't remediation. A failure webhook fires and then waits for a human. Nothing here reconciles inventories on its own, or launches an approved remediation template when an incident elsewhere calls for it.

That's the boundary of any DIY setup, not a knock on AWX. Part three picks up exactly there: CloudThinker connects to AWX read-only, reads every failed job the way an operator would, reconciles inventories continuously, and proposes approved templates as remediation, with nothing launched until a named person approves it. Or start now: Try CloudThinker free with 100 premium credits and no card required.