CircleCI Automation: Build Triage with Native Tools (Insights, Config, and the v2 API)

Everyone on your team has seen the flaky-tests tab. Far fewer teams have a quarantine process, meaning an actual decision about what happens to the test the tab names. That gap is where CircleCI automation succeeds or stalls. The platform is very good at telling you which test fails intermittently, and says nothing about whether to fix it, quarantine it or delete it.

Part one of this series mapped the waste patterns behind most CircleCI bills: flaky tests rerun into a credit sink, oversized resource classes, missing caches, and fan-out queueing on concurrency limits. This part is the hands-on version. It triages build failures and tunes workflows with only what CircleCI ships: Insights and its API, config.yml primitives, rerun with SSH, and the v2 API with curl. Budget an afternoon for a first pass.

We'll go tool by tool, because that's how the work actually happens.

01Tool 1

Insights

Duration, success rate, credits and flaky tests, in the app and the API.

02Tool 2

Test results

JUnit uploads that feed flake detection and timing-based splitting.

03Tool 3

Cache keys

Lockfile checksums so restore_cache hits when nothing changed.

04Tool 4

Approval jobs

The human gate before production. It stays human.

05Tool 5

Rerun with SSH

A shell in the exact container that failed.

06Tool 6

v2 API

Pipelines, workflows and jobs across every project, scripted with curl.

About tokens. Personal API tokens (User Settings → Personal API Tokens) carry your full user permissions, and they're what the v2 API expects. Project API tokens with the Read Only scope only work against the v1.1 API, so for v2 reporting scripts, create the personal token on a dedicated service user whose access you keep narrow. The examples use $CIRCLE_TOKEN and the slug gh/your-org/your-repo.

---

Tool 1: Insights, the dashboard and the API

Insights is the map. It answers "which workflows are slow, which are unreliable and which tests flake" without you instrumenting anything. Open it at your project → Insights (docs).

The three views that matter

• Workflow duration and success rate. Sort by median duration and by success rate. A workflow under 90% success that isn't gated on an external service is a triage target: someone is rerunning it to green and moving on.
• Per-job credits and duration. The job breakdown shows which jobs consume the most compute. A job whose p95 is far above its median is either short on resources or racing a flaky dependency.
• Flaky tests. Tests that passed and failed on the same commit, which is the cleanest flake signal there is. It needs uploaded test results (Tool 2). Without them the tab is empty.

Pull the same data from the API

The dashboard is fine for a look. For reporting you want the API. Summary metrics for every workflow in the project over 90 days:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/workflows?reporting-window=last-90-days" \
  | jq '.items[] | {name, success: .metrics.success_rate, p95: .metrics.duration_metrics.p95, credits: .metrics.total_credits_used}'

Flaky tests have their own endpoint:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/flaky-tests" \
  | jq '.flaky_tests[] | {test: .test_name, file: .file, job: .job_name, times: .times_flaked}'

How to read it. Sort by times_flaked. The top few tests are almost always behind most of your reruns and a large share of wasted credits. That ranked list is your triage backlog. Note what it is, though: names, not verdicts. Deciding what happens to each test is the manual loop this article ends on.

---

Tool 2: Test results and test splitting

Insights can only name flaky tests if you upload results. Emit JUnit XML and store it:

jobs:
  test:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run:
          command: npm test -- --reporters=jest-junit
          environment:
            JEST_JUNIT_OUTPUT_DIR: ./test-results
      - store_test_results:
          path: ./test-results

store_test_results powers both the flaky-tests view and timing-based test splitting (docs).

Split by timing, not by count

The common mistake is splitting tests across containers by file count, which leaves one container carrying every slow suite. Split by historical timing instead. CircleCI keeps it from your uploaded results:

jobs:
  test:
    parallelism: 4
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run: |
          TESTFILES=$(circleci tests glob "test/**/*.test.js" | \
            circleci tests split --split-by=timings)
          npm test -- $TESTFILES --reporters=jest-junit
      - store_test_results:
          path: ./test-results

How to read it. Raise parallelism until the median job duration stops falling. Past that point you pay for containers that finish early and sit idle, and you add to the concurrency queue from part one. The per-job duration view shows where that point is (test splitting docs).

---

Tool 3: Cache keys that actually hit

A cache that never hits is worse than no cache: you pay to save it and pay again to rebuild. The cause is almost always a key that changes every run. Derive it from a lockfile checksum:

steps:
  - checkout
  - restore_cache:
      keys:
        - deps-v1-{{ checksum "package-lock.json" }}
        - deps-v1-
  - run: npm ci
  - save_cache:
      key: deps-v1-{{ checksum "package-lock.json" }}
      paths:
        - ~/.npm

How to read it. The first key invalidates only when dependencies change. The second, partial key (deps-v1-) is the fallback, because a near-miss restore beats a cold install. Check the restore_cache step output on a recent run: "Found a cache" versus "No cache found" is the whole story. Bump v1 to force a clean rebuild when a cache goes bad (caching docs).

---

Tool 4: Approval jobs, the human gate you keep

Some gates shouldn't be automated, and the approval job is the primitive for them. It pauses a workflow until a person approves it in the UI:

workflows:
  deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - hold:
          type: approval
          requires:
            - test
      - deploy-prod:
          requires:
            - hold

How to read it. The hold job has no steps. Its only purpose is to stop the workflow and wait (approval job docs). Everything downstream is human-gated by design, and that matters for the rest of this series: when you automate triage later, the approval job stays a human decision. Automation should make the flaky-or-regression call obvious. It shouldn't click your production approve button.

---

Tool 5: Rerun with SSH, debugging the failure in place

When a job fails in a way you can't reproduce locally, don't push five commits of echo statements. Rerun the failed job with SSH from the job page (Rerun → Rerun job with SSH), then connect to the same container:

ssh -p 54782 12.34.56.78
# inside the container, inspect the real environment:
cat ~/project/test-results/*.xml
env | grep CIRCLE

How to read it. You land in the same image, working directory and environment variables the job saw (SSH debugging docs). It's the fastest way to separate an infrastructure failure (a missing binary, the wrong Node version, memory exhaustion) from a real test regression. The session holds a container and is billed like one, so disconnect when you're done.

---

Tool 6: The v2 API, custom reporting with curl

The dashboard shows one project at a time. For org-wide reporting you script the v2 API. The three endpoints you'll use most are pipelines, workflows and jobs (API reference):

# 1. Recent pipelines
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/project/gh/your-org/your-repo/pipeline" \
  | jq -r '.items[].id' | head -20

# 2. Workflows for one pipeline
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/pipeline/PIPELINE_ID/workflow" \
  | jq '.items[] | {name, status, id}'

# 3. Failed jobs in a workflow
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/workflow/WORKFLOW_ID/job" \
  | jq '.items[] | select(.status=="failed") | {name, job_number, type}'

How to read it. Chained together, these give you a failed-job feed across every project. Dump it to CSV and sort by frequency to get the org-level version of the flaky-tests list. You can also POST .../workflow/WORKFLOW_ID/rerun with {"from_failed": true} to rerun only the failed jobs. Be deliberate about that: rerunning flaky failures automatically is exactly the credit leak part one warned about.

---

One red workflow, triaged by hand

The nightly build-test workflow on api-service failed at 02:10. Before reading a single log line, name the three readings and what each predicts.

01Reading 1

Infrastructure

Fails in setup steps or with exit code 137. The commit is innocent.

02Reading 2

Flake

The test is already on the flaky list and passes on rerun at the same commit.

03Reading 3

Regression

First fails at one commit and fails again on rerun. Ping the author.

Check infrastructure first, because it's cheapest. The v2 job details give you the executor and timing, but step output still lives in the v1.1 job endpoint:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v1.1/project/gh/your-org/your-repo/4821" \
  | jq '.steps[] | {name, exit_code: .actions[0].exit_code, failed: .actions[0].failed}'

A failure in Spin up environment, checkout or restore_cache, or an exit code of 137 (killed, usually out of memory), means the commit is innocent. Here the failing step is npm test with exit code 1. Infrastructure is unlikely.

Then check the flake list. Filter the flaky-tests response from Tool 1 for the failing test:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/flaky-tests" \
  | jq '.flaky_tests[] | select(.test_name | test("refund"))'

If refunds › retries on gateway timeout shows up with times_flaked: 23, you're looking at a known flake. If it's absent, the next check matters more.

Finally, check for a regression. Recent runs of the workflow on main show where it started:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/workflows/build-test?branch=main" \
  | jq '.items[:10][] | {status, created_at, credits_used}'

A clean run of successes that turns red at one point, and stays red on rerun, is a regression. Match that run to its pipeline's commit and ping the author with the log excerpt.

That's three API calls and maybe 20 minutes for someone who knows where to look. It's also 20 minutes every time, and it still ends with a person deciding what to do with the test.

---

What each native tool answers, and what it leaves

Read down the right-hand column and the pattern is plain. The native tools are good at producing facts. Turning facts into a decision, on every red build, is still a person's job.

---

The honest part: where native triage stops

Run everything above. Know its limits before you call it a strategy.

It's point-in-time. The Insights window you read describes last week. This sprint's new tests and config changes start producing flakes and cache misses right away. Every pattern here is a process, not a one-time cleanup.

It costs engineer hours on repeat. The first pass is an afternoon. Every later pass competes with feature work, so "weekly triage" quietly becomes "when the build's been red for a day."

The verdict is still human. This is the real ceiling.

01Insights

Names the test

Flaky tests ranked by times flaked, with job and file.

02Engineer

Reads the logs

Step output, test history, the commit and recent config changes.

03Verdict

Fix, quarantine or delete

A judgment about your codebase that no native tool makes.

04Next red build

Repeat

Until the verdict is made, the test keeps rerunning and billing.

The flaky-tests endpoint names the test. It can't tell you whether that test guards a real race condition worth fixing, makes an assertion that was never deterministic and should be deleted, or should be quarantined while someone investigates. Every day that call sits unmade, the test keeps getting rerun and keeps billing.

---

What comes next

Part three puts that loop on a schedule. CloudThinker watches failed workflows through a read-only CircleCI token, reads the job logs and test results, classifies infrastructure versus flake versus regression, correlates the failure with the commit and recent config.yml changes, and proposes the fix: a corrected cache key, a right-sized resource class, a quarantine PR.

It acts within Auto Mode, so you decide what runs on its own and what waits for approval. Approval jobs stay human.

To see it on your own project, start with the triage you just did by hand:

"Why did last night's build-test workflow on api-service fail? Tell me whether it was infrastructure, a known flaky test or the commit, and show me the log and test evidence."

Try CloudThinker free with 100 premium credits and no card required, or continue with part three.