Memory That Can Be Wrong: How Lessons From Past Incidents Get Used
Checkout latency spikes on a Tuesday. Three weeks ago the same symptom turned out to be connection pool exhaustion, and someone fixed it by raising the pool size. The on-call remembers that, raises the pool again, and waits.
Latency stays high. This time the cause was a slow query introduced by a deploy an hour earlier. The memory was real. It just wasn't about this incident.
The scenario is illustrative. The risk is real for any system that remembers. A lesson that's right once becomes a shortcut, and a shortcut skips the evidence. We built incident memory for Frontier Investigation Agents on one assumption: memory can be wrong, so it has to be treated that way.
Why remembering is harder than it looks
The obvious design is a lookup table. Symptom in, past fix out. It's fast, and it fails the way the on-call above failed.
Three things break it.
Symptoms repeat, causes don't. High latency, 5xx spikes and restarting pods look the same across very different failures. Matching on the symptom recalls the wrong incident with full confidence.
One incident isn't a pattern. A fix that worked once might have worked by accident. Promote it to a rule after a single case and you've encoded a coincidence.
Fixes go stale. A lesson written before a migration can be actively harmful after it. Without a way to age out or contradict old notes, memory only grows, and the oldest notes carry the same weight as the newest.
Two kinds of memory
CloudThinker keeps two layers.
Workspace memory is what agents retain between conversations: environment notes, connection notes, how each teammate likes to work, and a nightly recap of the day's conversations. It belongs to the workspace, so what one engineer teaches an agent helps everyone.
Incident memory is the incident-focused part of workspace memory. It holds lessons written during root-cause analysis and read at the start of a later investigation. It's narrower on purpose: it exists to help the next diagnosis, and nothing else.
That last point is a hard boundary. Pulse, the Detect stage of Resolve that classifies incoming events, may keep its own cluster memory, but it never reads incident-memory files. A past lesson can shape how an incident gets investigated. It can't change how a new alert gets classified.
What gets written
Incident memory is agent-written. CloudThinker doesn't extract a fixed record from every incident. During an investigation, the agent writes only what it judges useful for a later one.
01Observation
One fact
A concrete fact or action from one investigation.
02Recollection
One incident
What happened in a specific past incident and what changed.
03Lesson
A pattern
A reusable pattern, backed by more than one source incident.
The three kinds encode how much weight a note deserves. An Observation is a fact from one investigation. A Recollection is what happened in one specific past incident and what changed. A Lesson is a reusable pattern, and it needs more than one source incident behind it.
That rule is the answer to "one incident isn't a pattern". A single case can become a recollection. It can't become a lesson.
Entries can carry the affected service, the evidence, the action taken, a verification command and an outcome. The evidence field matters most: the thing that separated the root cause from the symptom last time is what makes a memory useful the next time.
How recall works
Recall happens before hypothesis work, not after.
01Write
During an investigation
The agent saves only what it judges useful for a later one.
02Recall
Before hypotheses
Valid, recent entries load into context, capped and ordered by recent updates.
03Compare
Lead, not proof
Each recalled lesson is tested against current evidence like any other explanation.
04Outcome
Verified or not
A runbook verification command can confirm or fail the entry.
At the start of a new investigation, CloudThinker reads valid, recent entries from the workspace memory tree. Those entries land in the agent's context before it forms any hypotheses. Then the agent compares current evidence against them.
The order is deliberate. If memory arrived after the hypotheses, it would only confirm or reject them. Arriving first, it widens the set: "last time this looked like pool exhaustion" becomes one explanation to test, next to "a deploy changed a query" and whatever else the evidence suggests. The method behind that is in Hypothesis First.
A recalled lesson is a lead, never proof. The past resolution has to survive the same evidence check as every other hypothesis. In the Tuesday scenario, the pool lesson would be tested against pool metrics, found not to fit, and recorded as ruled out.
Recall is also bounded. It's capped and ordered by recent updates, so the agent never reads the entire history. Only entries that pass the memory format and safety checks are eligible at all.
Verified, Unverified, Failed
A memory without an outcome is a story. So each incident-memory entry can carry one of three.
01Verified
It held
A follow-up check confirmed the expected result.
02Unverified
No result yet
No conclusive follow-up result is recorded.
03Failed
Counter-evidence
The expected result did not hold. Never offered as guidance again.
When a runbook includes a verification command, the agent can schedule one follow-up check and update the outcome. A runbook step might carry something as plain as this:
kubectl rollout status deployment/checkout -n production
If the check confirms the expected result, the entry is Verified. If no conclusive result is recorded, it stays Unverified. If the expected result didn't hold, it's Failed.
Failed is where most memory systems go quiet. Ours doesn't. A failed lesson is never presented to the agent as valid guidance. A failed recollection can stay as counter-evidence, so a later investigation doesn't repeat the same assumption. "We tried raising the pool, and it didn't help" is useful the next time someone reaches for the pool.
The design decisions
Agent-written, not auto-extracted. Dumping every incident into memory floods recall with noise. Letting the agent decide what's worth keeping means fewer, sharper entries.
Lessons need more than one incident. This is the cheapest guard against encoding coincidence, and it costs nothing at write time.
Failure is kept, not deleted. Deleting a failed fix loses the most useful fact in it. Kept as counter-evidence, it stops the same wrong turn.
Recall before hypotheses. Memory widens the search instead of closing it.
Nothing is forever. Workspace memory that no agent has used for 90 days is archived and stops informing conversations. Stale context ages out even when nobody cleans it up.
People can correct it. Memory is agent-maintained, so you fix it the same way it was written: tell any agent what's wrong ("forget that we exempt the staging cluster"), and the correction applies to every later conversation in the workspace.
Honest limits
- The incident UI doesn't show an "informed by similar incidents" badge or a list of recalled memories. Memory changes the context behind the scenes, so you won't see which lessons were read.
- Verification isn't universal. A follow-up check runs only when a runbook includes a verification command. Without one, an entry stays Unverified.
- Re-running an investigation doesn't mechanically replace an older entry. The agent can revise, combine or supersede memory as evidence changes, but it's a judgment, not a rule.
- Incident memory has to be available for your workspace before entries are written or recalled.
- Workspace memory is shared, not private. Keep personal or sensitive details out of chat, and use a credential reference instead of pasting a secret. More on that in What Never Reaches the Model.
Try it
Add a verification command to one remediation runbook, so the next fix that uses it can earn an outcome. Then ask:
"What do you remember about checkout latency incidents, and which of those lessons were verified?"
"Forget the note that says raising the connection pool fixes checkout latency. It didn't hold last time."
"Investigate this incident, and tell me which past incidents you compared it against and why they did or didn't fit."