Better Stack Integration: AI Agents From Alert to Root Cause
At 02:14 the checkout-api monitor fails from all four regions with HTTP 503. Better Stack does its part perfectly: it confirms the failure, opens the incident, and the on-call's phone rings within a minute. They open the incident and find a red monitor, a response-time graph and nothing else. The next 40 minutes are theirs to spend: which service sits behind the endpoint, what changed tonight, which logs to read.
Part one mapped where that toil survives even a well-run uptime and on-call stack. Part two pushed Better Stack's native features (monitors, heartbeats, escalation policies, webhooks, log alerts, status pages) as far as they go, and ended at an honest ceiling: escalation finds a human fast, but it doesn't do the investigation for them.
This part removes that ceiling. A CloudThinker Better Stack integration picks up the incident when it opens, investigates while the page is still ringing, and writes down what it found. The escalation chain stays exactly as you configured it. The on-call still gets paged and still decides. What changes is what they find on arrival: which service failed, what the logs said in the minutes before the check went red, what changed underneath, and a proposed fix waiting for approval.
Why webhooks and runbooks stop at the page
Part two's webhook receivers are useful, and they share one structural limit: each one runs the steps someone wrote before the incident happened. A receiver that posts a runbook link posts the same link whether the cause is a bad deploy, an expired certificate or a database failover. A first-response script that restarts the service restarts it whatever the evidence says, including when the restart hides the cause.
Real investigation branches. A 503 from every region means something different from a timeout from one region, and the next check depends on which you got. That's the part an engineer does at 3 a.m., and it's the part a fixed receiver can't do.
Connecting: OAuth, about two minutes, read-only by default
01Detection and paging
Better Stack
- Monitors and heartbeatsWhich check failed, from where, how
- Incidents and escalationTimestamps, acknowledgment, policies
- Telemetry logsThe window around the failure
OAuth, read-only by default
02Investigation
CloudThinker
KubernetesPod state, restarts, rollout history
Cloud accountTarget groups, certificates, scaling
DatabasesConnections, locks, failovers
Crosses the boundary behind the URL
03Where the team works
Chat and on-call
Slack or TeamsDiagnosis and approvals in the thread
Better Stack escalationUnchanged. Still pages who it paged
- JiraFollow-ups with the evidence linked
Nothing new to watch
The connection is OAuth against your Better Stack account: no API tokens to mint and rotate by hand, no software to install. You authorize CloudThinker from the Better Stack connection guide, pick the team to connect, and the first sync starts.
Access is read-only by default. At connection time CloudThinker can see monitors, heartbeats, incidents, escalation policies and logs, and change nothing. Acknowledging incidents, posting timeline comments or touching monitor configuration each need an explicit write grant. Your security review gets a short answer: on day one, this integration can only look.
The infrastructure behind your endpoints (Kubernetes, your cloud account, your databases) is connected separately, with its own read-only credentials. That's what lets the investigation cross the boundary Better Stack can't.
One incident, end to end
01Signal
Incident opens
A monitor confirms failure from several regions. The page goes out as usual.
02Context
Incident and logs
Failure type, regions, flap history, related heartbeats, errors in the window.
03Behind the URL
Infrastructure
Pods, nodes, target groups, certificates and recent deploys.
04Output
Diagnosis
A named likely cause, the evidence, and a proposed fix with rollback.
Back to 02:14. Here's the investigation, step by step.
1. Read what the failure type implies. A 503 from all four regions, with fast response times, says the load balancer answered and had nothing healthy behind it. That shapes the readings:
- The application is down. Pods crashed or failed readiness. Predicts: restarts or not-ready pods, errors in the app's own logs.
- A dependency failed. The app is up but can't reach the database or a downstream service. Predicts: pods running, logs full of connection errors.
- The edge broke. An expired certificate, DNS or a routing change. Predicts: TLS or timeout errors, often from some regions only, not a clean 503 everywhere.
The edge reading is already unlikely. CloudThinker checks the other two rather than guessing.
2. Pull the incident's own context. The incident record, the monitor's recent history and any related heartbeat:
curl -s "https://uptime.betterstack.com/api/v2/incidents/INCIDENT_ID" \
-H "Authorization: Bearer $BETTERSTACK_TOKEN" |
jq '.data.attributes | {name, url, cause, started_at, status}'
The monitor hasn't flapped in 30 days, so this isn't noise. No heartbeat went quiet beforehand.
3. Search the logs around the failure. In Better Stack Telemetry, errors for the service behind the endpoint from 01:45 to 02:20. The window starts before the incident on purpose. What comes back: no connection errors to the database, but repeated container restarts for payments starting at 02:09, five minutes before the check failed. That weakens the dependency reading and points at the app.
4. Check behind the URL. With the cluster connected, the agent looks at pod state:
kubectl get pods -n payments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
Every payments pod shows OOMKilled as the last termination reason, with three to five restarts each. Then the change history:
kubectl rollout history deployment/payments -n payments
A new revision rolled out at 01:50. The memory graph shows usage roughly doubling after it, against an unchanged limit.
5. Write the finding and propose. The diagnosis in chat reads: "checkout-api went unreachable at 02:14 because the payments pods were OOM-killed after the 01:50 rollout doubled memory use. No database or edge involvement." It links the incident, the log query, the pod events, the memory graph and the rollout. The proposal is to roll back to the previous revision (kubectl rollout undo deployment/payments -n payments), with a follow-up ticket to raise the limit or fix the regression before re-deploying.
6. Wait for a person, then verify. In Manual, the rollback waits for the on-call to approve it in the thread. After it runs, CloudThinker watches the monitor through Better Stack's recovery period and confirms the incident resolved and stayed resolved. The whole investigation is already the postmortem timeline.
That example is illustrative, but its shape is the common one: most of the value is correlation work (a monitor, to a log line, to a deploy) done every time, immediately, instead of only when a senior engineer has the patience.
You decide what runs
CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval, whether that's posting the diagnosis as an incident comment or rolling back a production release. Every step is logged: what was found, what was proposed, who approved, what changed and when. A post-incident review reads the trail instead of rebuilding it from Slack.
The escalation chain stays intact
It's the first question every on-call lead asks, so plainly: nothing about your Better Stack escalation changes. Your policies still fire, your on-call calendar still decides who gets paged, and your status page still updates the way you wired it in part two. CloudThinker doesn't stand between the incident and the human. It works in parallel with the page. Time to acknowledge is still whatever your escalation policy makes it. Acknowledge to resolve is what shrinks.
What the first week typically surfaces
Illustrative numbers, a composite of what tends to turn up in the first week on a mid-market setup with 60 to 80 monitors. Your account will differ.
| Finding | Detail | Why it matters |
|---|---|---|
| Recurring root cause behind "random" downtime | 4 of the last 9 incidents traced to the same connection-pool exhaustion | One fix removes ~45% of recent pages |
| Flapping monitors | 3 monitors with confirmation periods too short for a slow upstream | Each flap is a page; tuning ends the noise |
| Silent heartbeat coverage gap | 2 nightly jobs with heartbeats created but never wired to an escalation policy | Failures were invisible until a customer noticed |
| Deploy-correlated incidents | 5 incidents opened within 10 minutes of a deploy, never linked to it | Root cause was in the release, not the infrastructure |
| Log errors preceding downtime | Error-rate ramp visible in logs 20+ minutes before checks failed | Earlier, quieter signal than the uptime check |
Why this beats the traditional approach
| Escalation plus webhooks | With CloudThinker | |
|---|---|---|
| What the on-call opens | A red monitor and a graph | A diagnosis with evidence and a proposed fix |
| Unfamiliar failures | Same receiver steps every time | Investigated from the evidence |
| Behind the endpoint | Out of reach for uptime checks | Pods, cloud resources and databases checked |
| Change correlation | From memory, if at all | Deploys checked against the failure window |
| Postmortem timeline | Rebuilt the next day | Written during the investigation |
The deeper difference is where the knowledge lives. In the traditional setup, knowing that api-prod-eu is really the payments service, and that its memory limit is tight, sits in two people's heads. CloudThinker writes that down with every investigation, so the next incident on the same service starts from what the last one learned.
Getting started
Connect Better Stack, then connect the infrastructure behind your most important monitor. Ask in plain language. Each answer cites its sources (the incident record, the log query, the infrastructure evidence) so you verify rather than trust.
"Why did the checkout-api monitor go down at 02:14 last night? Pull the incident, the logs around it, and anything that changed on the cluster. Propose only, change nothing."
"Which of our Better Stack monitors flapped more than three times this month, and what do they have in common?"
"Summarize last week's incidents: root causes, time to acknowledge, time to resolve, and which ones shared a cause."
What the agents won't do
- Act beyond the policy you set. Read-only by default. No acknowledgment, comment or fix unless you've granted write access and Auto Mode lets it run.
- Silence or reroute your pages. Escalation policies, on-call schedules and status pages stay under Better Stack's control and yours.
- Leave gaps in the record. Every investigation and action is in the audit trail with its evidence and its approver.
- Make your judgment calls. Naming the likely cause is CloudThinker's job. Deciding to roll back a release at 3 a.m. is an engineering decision, made by an engineer who now has the evidence.
Related reading
- From page to root cause with PagerDuty
- Auto Mode: how CloudThinker gates every change
- MTTR vs MTTD, MTTA and MTBF
- CloudThinker Connections
From paged to informed
Parts one and two gave you a well-tuned Better Stack: fast detection, reliable escalation, logs and status pages in place. The gap left was never detection. It was the investigation that starts after the page. Teams that put CloudThinker on that gap typically cut incident resolution time by 40 to 60%, not because anything detects faster, but because the human arrives at minute one with the context that used to take 40 minutes to assemble.
Try CloudThinker free with 100 premium credits and no card required, then follow the Better Stack connection guide and let the next incident open with a diagnosis already written.
