Jenkins Build Failure Analysis Using Only Native Tools

Paste these three lines into your Script Console (Manage Jenkins → Script Console, admin required) before you read anything else:

// Minutes the oldest queued item has been waiting
println(Jenkins.instance.queue.items.collect { ((System.currentTimeMillis() - it.inQueueSince) / 60000) as int }.max() ?: 0)

// Builds that have been running for more than an hour, right now
println(Jenkins.instance.getAllItems(Job).count { it.isBuilding() && System.currentTimeMillis() - it.lastBuild.startTimeInMillis > 3600000 })

// Jobs with no build discarder (unbounded disk growth)
println(Jenkins.instance.getAllItems(Job).count { it.buildDiscarder == null })

If any of those numbers surprised you, this guide is for you. Effective Jenkins build failure analysis doesn't require buying anything. The controller already knows why your queue is deep, which builds are stuck and where the disk went. It just doesn't volunteer the information. This is a hands-on tour of extracting it with native tooling only: the Script Console, the log-reading UIs, the JUnit plugin, build discarders, the REST API, and Jenkinsfile wrappers done properly.

This is part two of a three-part series. Part one mapped the failure and waste patterns that dominate Jenkins installations: flaky tests, executor starvation, error-swallowing pipelines, plugin drift. Part three covers running this triage continuously with an AI agent.

Budget 2 to 4 hours for a first pass on a controller with 100+ jobs.

01Tool 1

Script Console

The only native surface that sees the whole fleet: queue, stuck builds, disk, failure rate.

02Tool 2

Stage logs

Pipeline Graph View and Timestamper, so you read the failed stage instead of the whole log.

03Tool 3

JUnit trends

Per-test history that separates a flake from a permanent failure from a slow agent.

04Tool 4

Discarders

Log rotation and workspace cleanup, so the controller stops filling its own disk.

05Tool 5

REST API

Every page has a JSON twin. Scripts ask the questions you keep clicking for.

06Tool 6

Wrappers

retry and timeout in the right order, retrying agents and never tests.

The six native tools this guide covers. All of them ship with Jenkins or a standard plugin.

The loop you're trying to make cheaper

Every tool below serves one loop. A build goes red, someone finds the failing stage, checks whether the commit or the test has a history, decides which of the three readings from part one applies (infrastructure, known flake, real regression), and routes it.

01Locate

Failed stage

Open the red stage in the graph view and pull its log with timestamps.

02History

Same SHA?

Check whether the commit passed elsewhere and whether the test has failed before.

03Judgment

Classify

Infrastructure, flake or regression. No native tool makes this call for you.

04Act

Route

Node to platform, flake to quarantine, regression to the author.

The native triage loop. Every step except the third can be scripted with the tools below.

Keep that loop in mind as you read. Each tool either shortens a step or removes a reason to do it at all.

Tool 1: the Script Console, fleet-wide truth in Groovy

Everything in the Jenkins UI is scoped to one job or one node. The Script Console is the only native surface that sees the whole fleet at once, which makes it your first stop, not your last resort. It runs with full controller privileges, so stick to read-only queries and never paste code you haven't read.

Queue analysis. A deep queue is either a capacity problem or a label problem, and why tells you which:

Jenkins.instance.queue.items.each { i ->
  def mins = ((System.currentTimeMillis() - i.inQueueSince) / 60000) as int
  println "${i.task.fullDisplayName}: ${mins} min, ${i.why}"
}

"Waiting for next available executor" across the board means you're short on agents. "There are no nodes with the label X" means someone renamed a label or a Jenkinsfile demands an agent that no longer exists. That build will queue forever while looking like a capacity issue.

Stuck and marathon builds. Builds holding an executor for hours are usually waiting on something that will never answer (a dead input step, a hung network call with no timeout):

Jenkins.instance.getAllItems(Job).each { job ->
  def b = job.lastBuild
  if (b?.isBuilding()) {
    def mins = ((System.currentTimeMillis() - b.startTimeInMillis) / 60000) as int
    if (mins > 60) println "${job.fullName} #${b.number}: running ${mins} min"
  }
}

Disk pressure per node. Agent workspaces fill quietly until builds start failing with errors that look nothing like "disk full":

import hudson.node_monitors.DiskSpaceMonitor

Jenkins.instance.computers.each { c ->
  def ds = DiskSpaceMonitor.DESCRIPTOR.get(c)
  println "${c.name ?: 'built-in'}: ${ds?.gbLeft ?: '?'} GB free"
}

Failure rate over the last 24 hours. One number that tells you whether today is a normal day:

def since = System.currentTimeMillis() - 86400000
def total = 0, failed = 0
Jenkins.instance.getAllItems(Job).each { job ->
  job.builds.byTimestamp(since, System.currentTimeMillis()).each { b ->
    total++
    if (b.result == Result.FAILURE) failed++
  }
}
println "Last 24h: ${failed}/${total} builds failed"

Run these weekly and paste the output into a doc. That's a health baseline most teams never have.

Tool 2: reading pipeline logs without drowning

The classic console log for a parallel pipeline interleaves every branch into one stream, which is hard to read past a few thousand lines. Two native fixes:

Use a stage-oriented view. Blue Ocean renders each stage and parallel branch as its own log pane, so you click the red stage and read only its output. Blue Ocean is in maintenance mode. On newer controllers, install its designated successor, the Pipeline Graph View plugin, which does the same job and is actively developed. Either way the workflow is the same: never start from the full console log. Start from the failed stage.

Timestamp everything. The Timestamper plugin prefixes every log line with a clock. In a declarative Jenkinsfile:

pipeline {
  agent any
  options {
    timestamps()
  }
  ...
}

Timestamps turn "the build is slow" into "the npm install step took 14 minutes at 03:12," which is the difference between a hunch and a diagnosis. Gaps between consecutive timestamps are where your build time actually goes. A 10-minute silence usually means a network wait or a lock, not compute.

When you must grep the raw log, pull it as text rather than scrolling the UI (BUILD_URL/consoleText, more on the API below), then search from the bottom up. The first error printed is often noise, and the line that killed the build is near the end.

Tool 3: the JUnit plugin, trends and flaky-test spotting

If your pipelines don't publish test results, you're triaging blind. The JUnit plugin ingests XML reports from effectively every framework (Surefire, pytest, Jest via reporters, go test via converters):

post {
  always {
    junit testResults: '**/build/test-results/**/*.xml', skipPublishingChecks: true
  }
}

Put it in post { always }, not as a build step, so results are captured even when the stage fails. That's exactly when you need them.

Once results flow, the job page grows a Test Result Trend chart. Read it like this:

Flat red line at a constant count. A permanently failing test that everyone has learned to ignore. Delete it or fix it; it's training your team to skip red builds.
Red count that oscillates between 0 and a small number with no correlated code change. Flaky tests. Click into a failing test and open its History view, a pass/fail record across builds. A test that fails every 5th to 15th run on unchanged code is flaky, full stop.
Sudden jump in total test duration. Often the earliest visible sign of a resource-starved agent, before builds actually fail.

Flaky tests deserve a quarantine list, not a retry loop. Part one covers why retry-until-green is the most expensive non-fix in CI.

Tool 4: build discarders and workspace cleanup

Jenkins keeps every build's logs and artifacts forever unless told otherwise, and a controller that's low on disk gets slow, then flaky, then down. Two settings fix most of it.

Build discarders. In a declarative pipeline:

options {
  buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '10'))
}

For the jobs that lack one, the Script Console can audit (and fix) the whole fleet. This one writes, so run it deliberately:

import jenkins.model.BuildDiscarderProperty
import hudson.tasks.LogRotator

Jenkins.instance.getAllItems(Job).findAll { it.buildDiscarder == null }.each { job ->
  job.addProperty(new BuildDiscarderProperty(new LogRotator(-1, 50, -1, 10)))
  println "Discarder set: ${job.fullName}"
}

Pipelines that define options { buildDiscarder(...) } in the Jenkinsfile will override this on their next run. That's fine; the Jenkinsfile is the right home for it.

Workspace cleanup. Stale workspaces on agents are the other disk sink. The Workspace Cleanup plugin gives you cleanWs():

post {
  always {
    cleanWs(deleteDirs: true, notFailBuild: true)
  }
}

Reserve pre-build cleanup (cleanWs() as the first step) for jobs with proven cache-corruption problems. Wiping caches on every run trades disk for minutes of rebuild time.

Tool 5: the REST API, build metadata with curl

Every page in Jenkins has a machine-readable twin at /api/json (remote access API docs). With a user API token (your profile → Security → API Token), you can script the questions you keep asking the UI. The tree parameter keeps responses small:

JENKINS=https://jenkins.example.com
AUTH='you:11your-api-token'

# Every job whose last build failed
curl -s -u "$AUTH" "$JENKINS/api/json?tree=jobs[name,lastBuild[number,result,duration]]" \
  | jq -r '.jobs[] | select(.lastBuild.result == "FAILURE") | "\(.name) #\(.lastBuild.number)"'

# One build's metadata
curl -s -u "$AUTH" "$JENKINS/job/my-app/lastBuild/api/json?tree=number,result,duration,timestamp"

# Per-stage timing for a pipeline build (Pipeline Stage View plugin)
curl -s -u "$AUTH" "$JENKINS/job/my-app/123/wfapi/describe" \
  | jq '.stages[] | {name, status, durationMillis}'

# The last 200 lines of a failing build's log
curl -s -u "$AUTH" "$JENKINS/job/my-app/123/consoleText" | tail -n 200

Ten lines of shell around these endpoints gets you a daily "what failed overnight" digest in Slack. It's the single highest-leverage script most Jenkins teams never write.

Tool 6: retry and timeout wrappers, done right

retry and timeout are the most misused steps in the pipeline vocabulary. Two rules make them useful instead of harmful.

Rule 1: nesting order is a decision, not a detail.

// Per-attempt budget: each of 2 attempts gets its own 20 minutes
retry(2) {
  timeout(time: 20, unit: 'MINUTES') {
    sh './deploy-to-staging.sh'
  }
}

// Total budget: both attempts must finish inside 20 minutes combined
timeout(time: 20, unit: 'MINUTES') {
  retry(2) {
    sh './deploy-to-staging.sh'
  }
}

Per-attempt is usually what you want for infrastructure steps. Total budget is what you want when the stage must not delay the pipeline past an SLA.

Rule 2: retry infrastructure, never tests. Wrapping a test stage in retry(3) turns every flaky test into three times the compute and a green checkmark that means nothing. Modern declarative pipelines can say this precisely, retrying only when the agent died, not when the code failed:

stage('Integration tests') {
  options {
    retry(count: 2, conditions: [agent(), nonresumable()])
  }
  steps {
    sh './run-integration-tests.sh'
  }
}

That retries on agent disconnection (a spot instance reclaimed mid-build) and refuses to retry a genuine test failure. It's the correct default for every stage that runs on ephemeral agents.

One red build, triaged with native tools only

Here's the loop end to end on an illustrative case (names and numbers are made up, the steps are real). payments-service build #4812 fails in the Integration tests stage at 10:41. The author's commit only touched a README.

Three readings, three predictions. If it's infrastructure, the stage log should end in a node-level error and the agent should show disk or memory pressure. If it's a known flake, the failing test should have red entries in its JUnit history on unchanged code. If it's a regression, the same SHA should fail again on a different agent.

Check the stage log first. Pull the tail of the log and look for a node signature:

curl -s -u "$AUTH" "$JENKINS/job/payments-service/4812/consoleText" | tail -n 60

The tail shows a java.net.BindException: Address already in use inside CheckoutFlowIT, no disk errors, and a normal test summary. That makes pure infrastructure unlikely.

Check the test's history. On the build page, open Test Result → CheckoutFlowIT → History. It failed 9 times in the last 120 builds, on commits that touched nothing near checkout. That's the flake signature.

Check where it ran. The build metadata says which node it used:

curl -s -u "$AUTH" "$JENKINS/job/payments-service/4812/api/json?tree=builtOn,result,duration"

builtOn is linux-agent-3, and the Script Console disk check shows that agent at 4 GB free. Plenty, so disk isn't part of this story.

Verdict. A known flake, most likely a port collision between parallel test branches. It goes on the quarantine list with an owner, and the author's commit is cleared. Total time: about 15 minutes for someone who knew where to look, and every step needed a person to decide what to check next.

What each native tool answers, and what it doesn't

Tool	Answers	Doesn't answer
Script Console	Fleet state right now: queue, stuck builds, disk	Why it got that way, or what it looked like at 2 a.m.
Stage logs and timestamps	Which stage failed, where time went	Whether this failure is new or the ninth this month
JUnit trends	Which tests are flaky or permanently red	Which commit or node made them flaky
Discarders and cleanup	Keeps the controller and agents healthy	Nothing about failures; it's hygiene
REST API	Any build fact you can script	The classification; scripts fetch, people judge
Retry and timeout	Contains infrastructure blips and hangs	Real regressions, which it can hide if misused

The right column is the part a script can't fill. Each tool returns a fact. Turning facts into a verdict (infrastructure, flake, regression) and routing it is still a person reading three tabs.

Where DIY triage stops

Run all of this and you'll have something rare: a Jenkins installation you can actually see into. Be honest about what you still won't have.

01Wall 1

Symptoms

Trend charts say a test is flaky. They don’t say it shares a port with a parallel branch.

02Wall 2

Human judgment

Classifying each red build and correlating it with the commit and node is still manual.

03Wall 3

Point in time

The Groovy snippets describe this morning. Nobody runs them at 2 a.m.

Where native tooling stops. The middle wall is the expensive one, and it regenerates with every build.

The dashboards show symptoms, not causes. The trend chart says a test is flaky; it doesn't say the test shares a port with another parallel branch. The queue query says builds wait 40 minutes; it doesn't say which label to add capacity to.

Root-causing a red build is still human work. Someone still opens the log, tells an infra failure from a test failure from a real regression, correlates it with the commit and the agent's state, and decides. On a busy controller that's a rotating tax of several engineer-hours a week, paid by your most senior people.

The scripts are point-in-time. The Groovy snippets describe this morning. Nobody re-runs them at 2 a.m. when the queue actually melts down.

What comes next

Everything above (reading the failing log, classifying the failure, correlating it with the commit and the node, tracking deployments through stages) is the loop CloudThinker runs continuously against Jenkins over a read-only API token. Proposed fixes, like quarantining the flaky test or resizing the agent pool, wait for your approval. That's part three of this series.