Jenkins Build Failure Analysis: 6 Patterns Wasting Your CI Hours

The build went red at 2 p.m. The engineer looked at it for eleven seconds, typed "retry," and went back to their branch. It passed on the second run. Nobody looked at it again.

That retry button is the most expensive control in your CI system. Every click buys a green checkmark and sells a piece of information: why the build failed, whether it'll fail again tomorrow, and whether the thing that failed was your code, your tests, or your Jenkins controller quietly running out of disk.

Real Jenkins build failure analysis starts from an uncomfortable premise: most red builds in a mature Jenkins installation aren't caused by the commit that triggered them. They come from a handful of recurring failure and waste patterns. Each one has a distinct signature in the console log and a concrete way to detect it. This guide walks all six: what each looks like, why it develops, one detection step you can run today, and what it typically costs.

This is part one of a three-part series. Part two is a hands-on Jenkins build triage guide using only native tooling; part three covers automating build triage and deployment tracking with an AI agent.

01Tests

Flaky tests

Same commit, different result. Retries hide them and the population grows.

02Agents

Infra as build failure

Full disks, OOM kills and dropped agents reported as a red build on your commit.

03Pipelines

Swallowed errors

A stage fails inside, the build says SUCCESS, and the failure shows up in production.

04Controller

Plugin drift

One upgrade pulls new dependencies and jobs that ran for years start failing.

05Executors

Zombie jobs

Builds with no timeout hold an executor for days, waiting on something that never answers.

06Capacity

Unmeasured queues

Time in queue is never shown on the build page, so nobody trends it.

---

Every red build is one of three things

Before the patterns, the frame that makes them useful. When a build fails, there are only three readings, and each predicts something different about what you'll find:

01Reading 1

Infrastructure

The node or a dependency failed. The commit is innocent and a retry may land on the same broken agent.

02Reading 2

Known flake

A test with a history of passing and failing on unchanged code. Belongs in quarantine, not a retry loop.

03Reading 3

Real regression

Fails at one changeset and reproduces. The author needs to know now, with the evidence.

• Infrastructure predicts a log that ends in a node-level error (disk, memory, connection) and no assertion failure. The same SHA usually passes elsewhere.
• A known flake predicts a test that has failed before on unchanged code. Its JUnit history is a scatter of red among green.
• A real regression predicts a failure that starts at one changeset and reproduces on retry, with a compile error or a consistent assertion.

The retry button treats all three the same way. That's the root of everything below.

---

1. Flaky tests treated as retry-until-green

What it is. A test that passes and fails on the same commit is a flaky test. A team that answers flakiness with re-runs has made it policy: retries mask the signal, the flaky set grows, and eventually every red build is presumed innocent until retried twice.

Why it happens. Fixing a flaky test is unglamorous work with an unclear owner. Clicking retry takes one second and unblocks the merge. The incentives all point one way, until the day a real regression rides through on the third retry because everyone had stopped reading failures.

How to detect it. Same commit, different results. In the Script Console (Manage Jenkins → Script Console), print the last 50 builds of a job with their git SHA:

def job = Jenkins.instance.getItemByFullName('team/service-x/main')
job.builds.limit(50).each { b ->
  def sha = b.getAction(hudson.plugins.git.util.BuildData)
              ?.lastBuiltRevision?.sha1String?.take(8)
  println "#${b.number}  ${b.result}  ${sha}"
}

Any SHA that appears with both FAILURE and SUCCESS is a flake or an infrastructure failure. Either way, it's not a code problem. The JUnit plugin's Test Result Trend chart on the job page tells the same story visually: a sawtooth of small failure counts that never grows or shrinks is a stable population of flaky tests.

Typical impact. Teams with an entrenched retry culture commonly burn 2 to 5 engineer-hours per day across a mid-size org on re-runs and "is it real?" Slack threads, and every retry doubles the compute cost of that build. The bigger cost is trust: once under half of red builds are real, engineers stop reading any of them.

2. Infrastructure failures disguised as build failures

What it is. The build didn't fail. The agent did. A workspace filled the disk, the JVM was OOM-killed, or the node dropped its connection mid-build. Jenkins reports it the same way it reports a compile error: a red ball on your commit.

Why it happens. Agents accumulate workspaces, caches and Docker layers until the disk fills. Nobody owns node hygiene, because node failures look like someone else's test failures.

How to detect it. These failures have console log signatures that no code change produces: java.io.IOException: No space left on device, FATAL: command execution failed, Agent went offline during the build, or an abrupt log ending with no test output at all. For a fleet-wide disk check, use the Script Console again:

Jenkins.instance.computers.each { c ->
  def disk = c.monitorData['hudson.node_monitors.DiskSpaceMonitor']
  println "${c.name ?: 'controller'}: ${disk}"
}

Or in the UI: Manage Jenkins → Nodes shows free disk and free temp space per node, and the built-in monitors mark nodes offline below the threshold.

Typical impact. In installations that haven't audited for it, a meaningful minority of "build failures" (often 10 to 30% of all reds) are infrastructure, not code. Each one costs a triage cycle that concludes "not my problem" and a retry that may land on the same broken node.

3. Pipeline scripts that swallow real errors

What it is. A Jenkinsfile with sh 'deploy.sh || true', a catch block that logs and continues, or returnStatus: true whose return value nobody checks. The stage fails internally, the pipeline reports Finished: SUCCESS, and the failure surfaces days later as a production incident with a green build history.

Why it happens. Someone silenced a noisy but harmless step during an incident, the pattern got copy-pasted into the shared library, and now it guards steps that matter. It's the CI version of taping over a warning light.

How to detect it. Grep your pipeline code (Jenkinsfiles and shared libraries) for the three classic silencers:

grep -rn --include='Jenkinsfile*' --include='*.groovy' \
  -e '|| true' -e 'returnStatus: true' -e 'catch.*(ignored\|ignore)' .

Then audit each hit: is the exit code checked afterward, or thrown away? In console logs, the signature is a stack trace or ERROR line followed shortly by Finished: SUCCESS. The Pipeline syntax docs cover catchError and warnError, which mark the stage or build result properly instead of hiding it:

catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE') {
  sh './smoke-test.sh'
}

Typical impact. Low frequency, highest severity on this list. One swallowed deploy error typically costs hours of incident time and, worse, breaks the only promise CI exists to keep: green means good.

4. Plugin drift breaking builds after upgrades

What it is. Jenkins is a small core and several hundred plugins with interlocking dependency constraints. An upgrade of the core, or of one plugin that pulls new versions of its dependencies, changes behavior or APIs. Jobs that ran for years start failing with errors that mention no file you own.

Why it happens. Plugin updates get applied ad hoc (often during an unrelated incident), there's no staging Jenkins to rehearse them on, and version pinning is rare outside configuration-as-code setups.

How to detect it. The console log signature is a Java-level failure rather than a build-tool one: java.lang.NoSuchMethodError, ClassNotFoundException, or Unable to serialize appearing right after an upgrade date. To snapshot your plugin state for comparison before and after upgrades:

Jenkins.instance.pluginManager.plugins.sort { it.shortName }.each {
  println "${it.shortName}:${it.version}" + (it.hasUpdate() ? '  (update available)' : '')
}

Save that output with the date in the filename. A diff between two snapshots is the fastest answer to "what changed on the controller last week." Also check Manage Jenkins → Plugins → Installed plugins, which flags disabled and unavailable dependencies. The managing plugins guide documents pinning and dependency behavior.

Typical impact. Episodic: quiet for months, then an upgrade takes the whole pipeline fleet down for an afternoon. A controller-wide breakage typically costs several engineer-days across the org per event, plus the deploys frozen while it's diagnosed.

5. Zombie jobs consuming executors

What it is. Builds that will never finish but still hold an executor: a sh step waiting on a prompt that will never come, an integration test wedged on a dead dependency, an input step nobody will answer. With no timeout, they run until someone notices, often days later.

Why it happens. timeout wrappers are opt-in in scripted and declarative pipelines, and the default is forever. One wedged job on a four-executor node silently removes a quarter of that node's capacity.

How to detect it. List everything currently running and how long it's been at it:

Jenkins.instance.computers.each { c ->
  c.executors.findAll { it.busy }.each { e ->
    def hours = (e.elapsedTime / 3600000).round(1)
    println "${c.name ?: 'controller'}: ${e.currentExecutable} (${hours} h)"
  }
}

Anything measured in days is a zombie. The Build Executor Status widget on the Jenkins dashboard shows the same data one node at a time. The fix belongs in the pipeline: timeout(time: 60, unit: 'MINUTES') { ... } around every stage that touches the network. It's a baseline item in any set of Jenkins pipeline best practices.

Typical impact. Each zombie is one executor of pure waste: compute billed for nothing, and capacity taken from the queue. On cloud agents, a wedged job on an idle but billing VM is often tens to hundreds of dollars per incident. The queue delay it causes downstream is usually worth more.

6. Queue times nobody measures

What it is. The invisible metric. Engineers experience "CI is slow" as total wait (queue time plus build time), but Jenkins only puts build duration on the build page. A 12-minute build that queued for 25 minutes reads as a 12-minute build, and the capacity problem never makes it into a ticket.

Why it happens. Queue time isn't shown anywhere prominent, so it's never trended, so executor capacity gets negotiated by anecdote ("CI felt slow this sprint") instead of data.

How to detect it. Snapshot the queue right now, with reasons:

Jenkins.instance.queue.items.each { item ->
  def mins = ((System.currentTimeMillis() - item.inQueueSince) / 60000).round(1)
  println "${item.task.name}: ${mins} min, ${item.why}"
}

Waiting for next available executor recurring across the working day is your answer. For trends, Manage Jenkins → Load statistics charts online executors, busy executors and queue length over time. A queue line that regularly rides above zero during business hours means you're under-provisioned (or over-zombied, see pattern 5). For per-build queue time as data, the Metrics plugin records time in queue per build.

Typical impact. At mid-market scale, sustained queueing commonly adds 20 to 50% to effective CI latency during peak hours. That's the same cost as slow builds (context switching, batched merges, deploys pushed to tomorrow) with none of the visibility.

---

The six signatures at a glance

Pin this next to the console log. Most triage is matching what you see to one of these rows:

The last column is the one most teams skip. Every pattern on this list survives because it has no owner, and the retry button is how ownerless problems get deferred.

---

Why the retry culture persists

Each pattern above survives for the same structural reason: the cost of a red build is paid by whoever hits it, but the cost of diagnosing it is paid by whoever volunteers. Retrying is individually rational and collectively ruinous.

01Red

Build fails

A flake, a full disk or a real bug. Nobody knows which yet.

02Habit

Retry

Eleven seconds of looking, one click, a green checkmark.

03Lost

No record

The cause is never written down, so the pattern never gets an owner.

04Drift

Trust erodes

More reds clear on retry, so people stop reading any of them.

Flaky tests stay flaky because retries hide them. Infrastructure failures stay undiagnosed because they clear on re-run. Queues stay unmeasured because nobody's build page shows them. And the classification knowledge (which log lines mean disk, which tests are known flakes, which plugin broke what last spring) lives in two or three senior engineers' heads, so it leaves when they go on holiday.

The way out is to make failure analysis cheap and systematic instead of heroic. Every detection step in this guide runs today from the Script Console or a grep. Part two of this series turns them into a repeatable DIY triage workflow with Jenkins' native tools: log-reading technique, JUnit trend analysis, fleet-wide Groovy queries, and retry and timeout wrappers done right.

Make the triage continuous

Everything above is point-in-time. The zombie you kill today is replaced next week, and flake populations regrow. CloudThinker connects to Jenkins through a read-only API token and watches build results continuously: it classifies each failure as infrastructure, flake or real regression, correlates it with the commit and the node state, and proposes fixes that wait for your approval. Part three of this series shows how it works.

Related reading

• Jenkins Build Failure Analysis Using Only Native Tools (part two)
• Automated Jenkins Build Failure Analysis with CloudThinker Agents (part three)
• CircleCI automation guide, the same failure patterns on a hosted CI
• Auto Mode: graduated autonomy with safe defaults

Try CloudThinker free with 100 premium credits and no card required.