Product

Better Stack Integration: AI Agents From Alert to Root Cause

Part three of the Better Stack series: CloudThinker connects over OAuth (read-only by default) and starts investigating as soon as an incident opens, while the page is still going out. In the worked example it reads the failure type, the logs and the cluster behind the endpoint, then names the cause (an OOM after a deploy) and proposes a rollback for approval. Escalation is untouched, and every write goes through Auto Mode.

8 min read
Cover Image for Better Stack Integration: AI Agents From Alert to Root Cause

Better Stack Integration: AI Agents From Alert to Root Cause

At 02:14 the checkout-api monitor fails from all four regions with HTTP 503. Better Stack does its part perfectly: it confirms the failure, opens the incident, and the on-call's phone rings within a minute. They open the incident and find a red monitor, a response-time graph and nothing else. The next 40 minutes are theirs to spend: which service sits behind the endpoint, what changed tonight, which logs to read.

Part one mapped where that toil survives even a well-run uptime and on-call stack. Part two pushed Better Stack's native features (monitors, heartbeats, escalation policies, webhooks, log alerts, status pages) as far as they go, and ended at an honest ceiling: escalation finds a human fast, but it doesn't do the investigation for them.

This part removes that ceiling. A CloudThinker Better Stack integration picks up the incident when it opens, investigates while the page is still ringing, and writes down what it found. The escalation chain stays exactly as you configured it. The on-call still gets paged and still decides. What changes is what they find on arrival: which service failed, what the logs said in the minutes before the check went red, what changed underneath, and a proposed fix waiting for approval.


Why webhooks and runbooks stop at the page

Part two's webhook receivers are useful, and they share one structural limit: each one runs the steps someone wrote before the incident happened. A receiver that posts a runbook link posts the same link whether the cause is a bad deploy, an expired certificate or a database failover. A first-response script that restarts the service restarts it whatever the evidence says, including when the restart hides the cause.

Real investigation branches. A 503 from every region means something different from a timeout from one region, and the next check depends on which you got. That's the part an engineer does at 3 a.m., and it's the part a fixed receiver can't do.


Connecting: OAuth, about two minutes, read-only by default

01Detection and paging

Better StackBetter Stack

  • Monitors and heartbeatsWhich check failed, from where, how
  • Incidents and escalationTimestamps, acknowledgment, policies
  • Telemetry logsThe window around the failure

OAuth, read-only by default

02Investigation

CloudThinkerCloudThinker

  • KubernetesKubernetesPod state, restarts, rollout history
  • AWSCloud accountTarget groups, certificates, scaling
  • PostgreSQLDatabasesConnections, locks, failovers

Crosses the boundary behind the URL

03Where the team works

Chat and on-call

  • SlackSlack or TeamsDiagnosis and approvals in the thread
  • Better StackBetter Stack escalationUnchanged. Still pages who it paged
  • JiraFollow-ups with the evidence linked

Nothing new to watch

Better Stack sees the symptom. The cause lives in the systems behind the endpoint, which you connect separately.

The connection is OAuth against your Better Stack account: no API tokens to mint and rotate by hand, no software to install. You authorize CloudThinker from the Better Stack connection guide, pick the team to connect, and the first sync starts.

Access is read-only by default. At connection time CloudThinker can see monitors, heartbeats, incidents, escalation policies and logs, and change nothing. Acknowledging incidents, posting timeline comments or touching monitor configuration each need an explicit write grant. Your security review gets a short answer: on day one, this integration can only look.

The infrastructure behind your endpoints (Kubernetes, your cloud account, your databases) is connected separately, with its own read-only credentials. That's what lets the investigation cross the boundary Better Stack can't.


One incident, end to end

01Signal

Incident opens

A monitor confirms failure from several regions. The page goes out as usual.

02Context

Incident and logs

Failure type, regions, flap history, related heartbeats, errors in the window.

03Behind the URL

Infrastructure

Pods, nodes, target groups, certificates and recent deploys.

04Output

Diagnosis

A named likely cause, the evidence, and a proposed fix with rollback.

This runs in parallel with the page, so the evidence is there when the on-call opens the incident.

Back to 02:14. Here's the investigation, step by step.

1. Read what the failure type implies. A 503 from all four regions, with fast response times, says the load balancer answered and had nothing healthy behind it. That shapes the readings:

  • The application is down. Pods crashed or failed readiness. Predicts: restarts or not-ready pods, errors in the app's own logs.
  • A dependency failed. The app is up but can't reach the database or a downstream service. Predicts: pods running, logs full of connection errors.
  • The edge broke. An expired certificate, DNS or a routing change. Predicts: TLS or timeout errors, often from some regions only, not a clean 503 everywhere.

The edge reading is already unlikely. CloudThinker checks the other two rather than guessing.

2. Pull the incident's own context. The incident record, the monitor's recent history and any related heartbeat:

curl -s "https://uptime.betterstack.com/api/v2/incidents/INCIDENT_ID" \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" |
  jq '.data.attributes | {name, url, cause, started_at, status}'

The monitor hasn't flapped in 30 days, so this isn't noise. No heartbeat went quiet beforehand.

3. Search the logs around the failure. In Better Stack Telemetry, errors for the service behind the endpoint from 01:45 to 02:20. The window starts before the incident on purpose. What comes back: no connection errors to the database, but repeated container restarts for payments starting at 02:09, five minutes before the check failed. That weakens the dependency reading and points at the app.

4. Check behind the URL. With the cluster connected, the agent looks at pod state:

kubectl get pods -n payments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.reason}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

Every payments pod shows OOMKilled as the last termination reason, with three to five restarts each. Then the change history:

kubectl rollout history deployment/payments -n payments

A new revision rolled out at 01:50. The memory graph shows usage roughly doubling after it, against an unchanged limit.

5. Write the finding and propose. The diagnosis in chat reads: "checkout-api went unreachable at 02:14 because the payments pods were OOM-killed after the 01:50 rollout doubled memory use. No database or edge involvement." It links the incident, the log query, the pod events, the memory graph and the rollout. The proposal is to roll back to the previous revision (kubectl rollout undo deployment/payments -n payments), with a follow-up ticket to raise the limit or fix the regression before re-deploying.

6. Wait for a person, then verify. In Manual, the rollback waits for the on-call to approve it in the thread. After it runs, CloudThinker watches the monitor through Better Stack's recovery period and confirms the incident resolved and stayed resolved. The whole investigation is already the postmortem timeline.

That example is illustrative, but its shape is the common one: most of the value is correlation work (a monitor, to a log line, to a deploy) done every time, immediately, instead of only when a senior engineer has the patience.


You decide what runs

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval, whether that's posting the diagnosis as an incident comment or rolling back a production release. Every step is logged: what was found, what was proposed, who approved, what changed and when. A post-incident review reads the trail instead of rebuilding it from Slack.

The escalation chain stays intact

It's the first question every on-call lead asks, so plainly: nothing about your Better Stack escalation changes. Your policies still fire, your on-call calendar still decides who gets paged, and your status page still updates the way you wired it in part two. CloudThinker doesn't stand between the incident and the human. It works in parallel with the page. Time to acknowledge is still whatever your escalation policy makes it. Acknowledge to resolve is what shrinks.


What the first week typically surfaces

Illustrative numbers, a composite of what tends to turn up in the first week on a mid-market setup with 60 to 80 monitors. Your account will differ.

FindingDetailWhy it matters
Recurring root cause behind "random" downtime4 of the last 9 incidents traced to the same connection-pool exhaustionOne fix removes ~45% of recent pages
Flapping monitors3 monitors with confirmation periods too short for a slow upstreamEach flap is a page; tuning ends the noise
Silent heartbeat coverage gap2 nightly jobs with heartbeats created but never wired to an escalation policyFailures were invisible until a customer noticed
Deploy-correlated incidents5 incidents opened within 10 minutes of a deploy, never linked to itRoot cause was in the release, not the infrastructure
Log errors preceding downtimeError-rate ramp visible in logs 20+ minutes before checks failedEarlier, quieter signal than the uptime check

Why this beats the traditional approach

Escalation plus webhooksWith CloudThinker
What the on-call opensA red monitor and a graphA diagnosis with evidence and a proposed fix
Unfamiliar failuresSame receiver steps every timeInvestigated from the evidence
Behind the endpointOut of reach for uptime checksPods, cloud resources and databases checked
Change correlationFrom memory, if at allDeploys checked against the failure window
Postmortem timelineRebuilt the next dayWritten during the investigation

The deeper difference is where the knowledge lives. In the traditional setup, knowing that api-prod-eu is really the payments service, and that its memory limit is tight, sits in two people's heads. CloudThinker writes that down with every investigation, so the next incident on the same service starts from what the last one learned.


Getting started

Connect Better Stack, then connect the infrastructure behind your most important monitor. Ask in plain language. Each answer cites its sources (the incident record, the log query, the infrastructure evidence) so you verify rather than trust.

"Why did the checkout-api monitor go down at 02:14 last night? Pull the incident, the logs around it, and anything that changed on the cluster. Propose only, change nothing."

"Which of our Better Stack monitors flapped more than three times this month, and what do they have in common?"

"Summarize last week's incidents: root causes, time to acknowledge, time to resolve, and which ones shared a cause."


What the agents won't do

  • Act beyond the policy you set. Read-only by default. No acknowledgment, comment or fix unless you've granted write access and Auto Mode lets it run.
  • Silence or reroute your pages. Escalation policies, on-call schedules and status pages stay under Better Stack's control and yours.
  • Leave gaps in the record. Every investigation and action is in the audit trail with its evidence and its approver.
  • Make your judgment calls. Naming the likely cause is CloudThinker's job. Deciding to roll back a release at 3 a.m. is an engineering decision, made by an engineer who now has the evidence.

Related reading

From paged to informed

Parts one and two gave you a well-tuned Better Stack: fast detection, reliable escalation, logs and status pages in place. The gap left was never detection. It was the investigation that starts after the page. Teams that put CloudThinker on that gap typically cut incident resolution time by 40 to 60%, not because anything detects faster, but because the human arrives at minute one with the context that used to take 40 minutes to assemble.

Try CloudThinker free with 100 premium credits and no card required, then follow the Better Stack connection guide and let the next incident open with a diagnosis already written.

betterstackincidentresponseuptimeoncallaiagentcloudthinker