Why Incident Investigation Is Slow: Complexity, Silos and Scattered Knowledge
The payments incident in part one took three engineers 86 minutes to investigate. The failure behind it had already been analyzed in a postmortem six weeks earlier, and its early warning had been ignored forty times that month.
That's the uncomfortable follow-up this part takes on: good teams keep investigating problems they've already solved.
The question matters because the time adds up. In NeuBird's February 2026 survey of 1,039 SRE, DevOps and IT operations professionals, most said their teams spend 40% or more of their time on incident management. Google's original SRE guidance caps operational work at 50%, so engineers keep at least half their time for engineering. Many teams are now close to that line, and every repeat investigation pushes them closer.
My answer is structural. The complexity of our systems has outgrown what our organization charts can hold, and the knowledge needed to close an incident is rarely in the right place at the right time. Three things drive it: complexity, team silos and fragmented knowledge.
Complexity outgrew any one person's map
A decade ago, most enterprise systems were a handful of monoliths. Today they're hundreds of microservices running across several clouds, with dependency chains no single person understands end to end.
In many Vietnamese companies, especially in banking and telecom, that new architecture still lives alongside on-premise systems and a traditional core. The dependency chain crosses generations of technology as well as services.
So a symptom at the application layer can start almost anywhere:
- the database,
- the network,
- an infrastructure configuration,
- the core system,
- or a change another team shipped this afternoon.
The person holding the pager sees the symptom. The cause sits in a part of the map they don't own and may never have seen. That gap is what investigation is: walking the map until the symptom and the cause connect.
The three clocks from part one make this concrete. Detection takes minutes because monitoring is good at noticing that something is wrong. The fix takes minutes because once you know the cause, it's usually a rollback or a config change. Investigation takes the hour because it's the step that requires the whole map.
Silos turn investigation into coordination
Organizations manage complexity the sensible way: they split teams by service or by technical layer. Each team gets its own tools, its own dashboards and its own knowledge. That structure works well for building software.
It works badly during an incident, because incidents don't respect team boundaries.
01Application
Own dashboards
Its own logs, its own runbooks, its own view of the latency spike.
02Database
Own tools
Its own consoles and its own memory of what the pool did last time.
03Payments
Own process
Its own on-call rotation and its own idea of which deploy is risky.
04The incident
Three teams, two handoffs
It crosses every boundary above, and each crossing costs minutes.
Look at the 2:14 a.m. incident again. The on-call engineer didn't call the database engineer and the payments lead because they lacked skill. They called them because the knowledge they needed was spread across three teams. One team knew the latency pattern, one knew how the pool behaved under load, and one knew which afternoon deploy had touched the workers.
That means a large part of MTTI is really coordination time:
- Working out which team owns which component.
- Waiting for the owner to wake up and join.
- Handing over context, again, to each new person on the call.
None of those steps is investigation in the technical sense. Nobody is reading a trace while they wait for a callback. But the clock runs the whole time.
Coordination cost also grows with the system. Every new service adds an owner. Every new team adds a boundary. As both numbers rise, so does the number of handoffs per incident. This is why adding headcount or buying another tool for one team rarely moves MTTI for the organization. Each team gets better at seeing its own slice. The incident still has to cross the same lines, and someone still has to carry the context across.
Every engineer you add to an incident call adds cost. It rarely shortens the call by the same amount.
Fragmented knowledge is held in place by three mechanisms
The clearest symptom of all this is fragmented knowledge. The lesson exists somewhere. It just isn't available when it's needed. I see three mechanisms keeping it that way.
Knowledge lives in people. Most operational knowledge sits in the heads of the engineers who handled the last incident. When that person is on leave, has moved teams or works in another time zone, the organization loses access to what they know. It hasn't lost the knowledge. It has lost the route to it.
Postmortems are written for the archive. A postmortem is written to record what happened and to support the review afterwards. That's a reasonable goal, and it produces a document that's hard to search fast in the middle of an incident. At 3 a.m. nobody is reading six weeks of postmortems to see whether this latency pattern is familiar.
Prevention competes with the roadmap. Postmortem action items have to fight for capacity against product work. They usually lose, because the benefit of preventing something is hard to quantify in the short term. The connection pool fix from six weeks earlier is a typical casualty: written down, agreed, never scheduled.
01In people
Knowledge walks
The engineer who solved it last time is on leave, on another team or asleep.
02In archives
Postmortems sleep
Written for review after the fact, hard to search while the pager is going.
03In backlogs
Prevention waits
Follow-up items lose to the roadmap because their payoff is hard to show.
04In production
The failure repeats
The lesson never reached operations, so the same incident comes back.
These three aren't separate problems. They form a loop. The lesson stays with a person, or in a document, or in a backlog, and never makes it into how operations actually runs. So the incident repeats. Each repetition produces another postmortem and another action item, which join the queue behind the last ones.
Noisy alerts close the loop from the other side. An alert that fires forty times a month without leading to action has stopped being an alert. In the example, the earliest signal of the incident arrived more than an hour before the page, and it was ignored for exactly that reason. The knowledge to act on it existed. The signal and the knowledge just never met.
A metric for how fast you learn: time to remember
Standard incident metrics describe a single incident. MTTD, MTTI and MTTR all start and stop within one night. None of them tells you whether the organization learned anything from it.
So I propose adding one metric to the operational set: time to remember.
Time to remember runs from the first time a failure happens to the point where the organization can recognize that failure automatically, across the whole system.
In the 2:14 a.m. example, time to remember was already longer than six weeks, and still running. The lesson existed in a document. It hadn't reached operations. Nothing would have flagged the risky deploy at review, and nothing linked the 1:00 alert to the known failure.
What makes this metric useful is that it explains the others. When time to remember drops:
- the repeat rate drops, because known failures get caught earlier,
- investigation time drops, because the first hypothesis is the known one,
- and the share of team time spent on incidents drops with them.
You can't improve time to remember by working harder during incidents. You improve it by changing where knowledge lives: out of individual heads and archived documents, and into something that review, alerting and investigation all consult every time.
The minimum metric set to track every month
Time to remember works best alongside five other numbers. Together they're the minimum set I'd ask any operations organization to report every month.
01Incident share
Of ops time
How much of the team goes to incidents instead of engineering.
02Repeat rate
Known failures
How many incidents are a failure the organization has seen before.
03MTTI
Time to investigate
From alert to a confirmed root cause and a fix to apply.
04Actionable alerts
Signal, not noise
How many alerts led to someone doing something.
05Prevention done
Follow-ups closed
How many postmortem action items actually shipped.
06Time to remember
First seen to caught
From the first occurrence to the point the organization recognizes it everywhere, on its own.
| Metric | What it tells you | How to start measuring it |
|---|---|---|
| Incident share | How much engineering time goes to incidents | Ask teams to tag incident work, even roughly |
| Repeat rate | How often you meet a failure you've seen before | Link each new incident to past ones in the postmortem |
| MTTI | Where the hours actually go | Record when the root cause was confirmed, not only open and close |
| Actionable alert rate | Whether your alerts still mean anything | Track which alerts led to an action and which were dismissed |
| Prevention completion | Whether lessons turn into changes | Report open postmortem action items by age |
| Time to remember | How fast the organization learns | Date the first occurrence of each failure class and the day it was first caught automatically |
Most of these need nothing new to measure. MTTI is the exception worth calling out. Most ITSM systems only record when an incident opens and when it closes, so investigation time disappears into the total. One added timestamp, root cause confirmed, is enough to split the number.
Once you have the split, the morning report changes. "Payments degraded, resolved in 94 minutes" becomes something a CIO can act on: how much of that was investigation, whether it was a repeat, and how long the organization has known about it.
What this says about agents
If investigation is slow because knowledge is in the wrong place, faster typing won't fix it. Neither will a smarter chatbot that sees one tool at a time.
What would help is work that crosses team boundaries without a handoff, reads evidence from every tool in one pass and checks it against what the organization already learned. That's the job of an investigation agent working on a shared context with a company memory. It's also why that memory has to be correctable, a point I take seriously in incident memory that can be wrong.
Agents don't change the diagnosis in this post. Complexity, silos and scattered knowledge are organizational facts, and they'd still be there with any tool. What agents can change is the cost of each one: the handoff, the search through old postmortems, the prevention item nobody had time to write. The rest of this series is about how to do that safely, starting with the layer it runs on in the operational excellence layer.
For the full argument, start with the series overview.
Related reading
- Why root cause analysis takes hours
- Inside an investigation
- Incident memory that can be wrong
- Frontier Investigation Agents
- CloudThinker Resolve
Sources
- NeuBird AI, 2026 State of Production Reliability and AI Adoption Report, survey of 1,039 professionals, February 2026 (vendor-run survey).
- Google, Site Reliability Engineering, chapter 5, "Eliminating Toil". sre.google
The incident in this post is a composite scenario.