CircleCI Automation with AI Agents: Build Triage That Ends the Rerun Reflex

The nightly workflow goes red at 2 a.m. By the time someone sees it, the ritual runs on its own: open the failed job, scroll a few thousand lines of log, squint at the last stack trace, click Rerun workflow from failed. If it goes green, nobody looks again. If it stays red, the same person scrolls the same log a second time. Scroll, squint, rerun is how most teams do CircleCI automation, and it burns credits every time a flaky test gets a free retry.

Part one mapped the failure and waste patterns behind real CircleCI usage: flaky tests rerun into a credit sink, oversized resource classes, workflows with no caching, fan-out stalled on concurrency limits, and failed workflows nobody re-examines once the rerun goes green. Part two covered the DIY toolkit: Insights and its API, test splitting, cache keys that hit, approval jobs, rerun with SSH and the v2 API with curl.

Part two ended at its honest ceiling. Insights names the flaky test, but deciding whether to fix, quarantine or delete it is still a human loop, and that loop runs again on every red workflow. This part covers the layer that closes it: CloudThinker watches failed workflows continuously, classifies each one, tracks deployments across environments and proposes fixes, with your approval jobs left exactly where you put them.

---

Why the rerun reflex survives every cleanup

Teams know reruns are expensive. The reflex survives anyway, for structural reasons.

Classification is the expensive step, and nothing does it for you. Insights counts failures. It doesn't say whether a given failure is the registry timing out, a test that's flipped 30 times this month, or a real bug in the commit. A rerun is the cheapest way to find out, so people use it as a diagnostic.

The evidence is spread across four places. Step output lives in the job. Test history lives in Insights. The commit lives in the pipeline. The config change that actually caused the wave of failures lives in git. Joining them by hand takes 20 minutes, and the rerun button takes one second.

Verdicts don't persist. When someone does triage a flake, the conclusion lives in a Slack thread. Next week a different person meets the same test and starts from scratch.

Native automation can only rerun. The API's from_failed rerun is the one action you can script without judgment, so it's the one that gets scripted, which turns a reflex into a policy.

---

Connecting CircleCI: a read-only token, about five minutes

01CI/CD

CircleCI

• Pipelines and workflowsStatus, commits, approval gates
• Jobs and test resultsStep output, JUnit results, executor
• InsightsDurations, credits, flaky tests

Read-only API token

02Triage

CloudThinker

• Classify each failureInfrastructure, flake or regression
• CorrelateThe commit and recent config.yml changes
• Propose the fixQuarantine, cache key, resource class

Nothing added to config.yml

03Where the team works

Chat and code

• Slack or TeamsFindings and approvals in the thread
• GitHub or GitLabFixes arrive as pull requests
• JiraFlake follow-ups with evidence linked

Approval jobs stay human

CloudThinker connects with a CircleCI API token used only for reads, and nothing added to your config.yml. No orb, no shared context secrets, no write access at connection time. Because project tokens only cover the v1.1 API, use a personal API token, ideally from a service user with read access to the projects you want covered.

You paste the token and pick the org. The first scan uses the same v2 API endpoints you called in part two: GET /project/{project-slug}/pipeline, /pipeline/{id}/workflow, /workflow/{id}/job, and the Insights routes under /insights/{project-slug}/workflows. The CircleCI connection guide lists the exact token setup so your security review has the minimal grant on record.

---

What happens when a workflow fails

01Signal

Workflow fails

A workflow moves to failed. The agent picks it up before anyone clicks rerun.

02Classify

Read the evidence

Step output, exit codes, test results and the test failure history.

03Correlate

Commit and config

The triggering commit and any recent change to .circleci/config.yml.

04Output

Verdict and fix

What failed, which reading the evidence supports, and a proposed change.

Classify the failure

The core skill of build triage is classification, the one thing Insights can't do. When a workflow fails, the agent pulls the job's step output and test results and answers the question the red badge didn't:

• Infrastructure failure. A docker pull timing out against the registry, checkout failing on a network blip, an out-of-memory kill (exit code 137) from a resource class too small for the step, a cache restore that half-succeeded. The commit is innocent. The finding is about the container or the resources.
• Flake, and how flaky. A test that's passed for months and failed once isn't the same as one that's failed 30 times this month and passed on every rerun. The agent keeps per-test failure history across the project, which turns "flaky builds" from a feeling into a ranked list with a credit cost on each entry.
• Real regression. Compile errors, or assertion failures that start at exactly one commit and reproduce on rerun. The author should hear about it right away, with the evidence, and not after two hopeful reruns spent 15 minutes of a large resource class.

Each classification arrives with its evidence: the log excerpt, the test's failure history, the commit, and the resource class and duration of the failed job.

Correlate with the commit and recent config changes

A failure judged in isolation is half an answer. The agent checks each red workflow against the commit that triggered it and against recent changes to .circleci/config.yml. That's how the sneakiest part-one pattern gets caught: a wave of "test failures" that started the morning someone lowered a resource class, changed a cache key or reordered a requires. Instead of blaming whichever commit ran next, the agent points at the config diff that changed the behavior.

Track deployments across environments

Beyond failures, the agent follows delivery workflows job by job: which commit reached staging, which reached production, where the last deploy stalled and for how long, and whether a workflow is parked on an approval job everyone forgot about. If CircleCI is your deployment system of record, "what's actually in prod right now" becomes a question you ask in chat.

---

One nightly failure, end to end

The nightly build-test workflow on api-service fails at 02:10. Here's what the agent does, in the order it does it.

It names the readings first. Infrastructure would show as a failure in a setup step or an exit code of 137. A known flake would show the failing test on the flaky list, passing on rerun at the same commit. A regression would show the test passing on every run before one commit and failing on every run after.

It checks infrastructure. The failing step is npm test, exit code 1, on a medium Docker executor with memory well under the limit. Setup steps passed. Infrastructure is out.

It checks the flake history. The failing test is refunds › retries on gateway timeout. It isn't on the flaky list. It has passed on every run for 90 days. So it isn't a known flake.

It checks for a regression. The test started failing at the first nightly after a merge that changed the refund retry policy, and it fails again on a rerun at that commit. No config.yml changes in the window. That's the reading the evidence supports.

It writes the finding. In the team's Slack channel: the failing assertion, the 90-day pass history, the commit and its author, and a note that a rerun won't help. No rerun was spent. The author gets the message before standup instead of after two retries.

Flip one fact. If the test had flipped 23 times in a month and passed on rerun at the same commit, the finding would say "known flake," attach the credits it burned on reruns, and propose a quarantine PR for someone to approve. Same red badge, opposite right response. The difference is the classification step the rerun reflex skips.

---

Proposals before actions, approval jobs untouched

Triage produces findings, and findings imply fixes: quarantine a flaky test (a PR tagging it for exclusion until it's fixed), correct a cache key so restore_cache stops missing, right-size the resource class for a job that peaks at 8% of its memory, split a test job that's serializing your slowest feedback. CloudThinker acts within Auto Mode, so you decide which of those run on their own and which wait for approval.

One line matters more than the rest: approval jobs stay human. The agent reads the state of every type: approval gate and reports it, and it never clicks one. A deploy waiting on a person is a deploy the agent will surface and explain, never advance. Every finding, proposal, approval and action lands in an audit trail. If your team has been arguing about quarantine policy or automatic reruns in the abstract, this is where the argument gets settled with credit numbers instead of opinions.

---

What a first pass typically finds

The numbers below are illustrative, a composite for a mid-market setup: about 40 projects, a few hundred workflows a day, and credit spend nobody attributes per project. Yours will differ.

Nothing in that table is exotic. It's the part-one inventory, surfaced automatically, with each row carrying a classification and a proposed, approval-gated fix instead of a dashboard someone has to remember to open.

---

Why this beats the traditional approach

The deeper difference is memory. A person triaging builds starts from zero each time, and so does a rerun script. The agent keeps the history of every test and every classification, so the tenth failure of a flaky test gets handled in seconds instead of re-investigated.

---

Prompts to try in your first session

You direct CloudThinker in plain language from chat, and each answer cites the CircleCI data behind it (the same pipeline, workflow, job and Insights responses you pulled by hand in part two) so you can verify instead of trust:

"Why did last night's nightly workflow on api-service fail? Tell me whether it was infrastructure, a known flaky test or the commit, and show me the log and test evidence."

"Rank the flakiest tests across all projects over the last 30 days by credits burned on reruns. Which would you quarantine first, and why?"

"Which jobs run on a resource class larger than they need, and what would right-sizing them save in credits per month? Propose only, change nothing."

"Walk me through the deploy workflow: which commit is in staging and which in production, and is anything parked on an approval job right now?"

---

What the agent won't do

"AI with access to CI" should raise exactly these questions:

• Read-only by default. The connection token is used for reads only. Quarantining a test, editing a cache key or changing a resource class needs a scoped write grant, and the change still goes through Auto Mode.
• No reruns to hide real failures. The agent won't paper over a regression with reruns to turn the board green. A rerun is proposed only when the failure is provably infrastructure, the opposite of the reflex part one warned about.
• Approval jobs stay yours. The agent reads every type: approval gate and never advances one.
• No invisible changes. Every proposal and execution is in the audit trail with its evidence and approver. If it isn't in the trail, it didn't happen.

---

From red workflows to answers

Part one showed which patterns burn credits. Part two showed how far Insights and the v2 API take you, and that the verdict on each failure stays human. That verdict, made on every red workflow with the history of every test behind it, is what the agent layer adds. Teams that move from the rerun reflex to continuous, approval-gated triage typically cut time to an answer on a red workflow from tens of minutes to under one, and keep the credits the reflex was spending every night.

Try CloudThinker free with 100 premium credits and no card required, and follow the CircleCI connection guide to see your own first findings within the hour.