How To

Better Stack Integration Guide: Where Incident Toil Still Lives

Part one of the Better Stack series: detection and paging are solved, but five gaps still eat on-call time after the acknowledgment: the context hunt, unwired heartbeats, logs in another tab, timelines rebuilt by hand, and silent status pages. Each comes with an Uptime API call or console check to find it in your own account.

9 min read
Cover Image for Better Stack Integration Guide: Where Incident Toil Still Lives

Better Stack Integration Guide: Where Incident Toil Still Lives

Your phone buzzes at 3:07 a.m. You acknowledge the page in forty seconds. You know which service is actually broken forty minutes later. Everything in between (scrolling the monitor list, searching the deploy channel, opening log tabs, setting time ranges by hand) is toil, and it happens on every incident.

That gap is what this Better Stack integration series is about. Better Stack is a good stack: uptime checks from multiple regions, heartbeat monitors for cron jobs, on-call schedules with escalation policies, log management and status pages under one roof. Detection and paging are solved problems. But between "a human has been paged" and "the incident is resolved", almost everything is still manual, and that manual middle is where on-call hours actually go.

This guide maps the five places incident toil survives a well-configured Better Stack setup: what each looks like, why it persists, how to spot it in your own account today, and what it typically costs in minutes per incident.

01Seconds

Detect

Multi-region checks confirm the failure and open the incident.

02Under 2 min

Escalate

The escalation policy finds the on-call and they acknowledge.

0320 to 40 min

Orient

Which service, what changed, which logs. All by hand, half asleep.

04Minutes

Fix

Often one command once the cause is known.

051 to 2 hours

Record

The timeline is rebuilt the next day for the postmortem.

Typical ranges. Tooling solved the first two steps. Most of the hours sit in the third and fifth.

Part two is a hands-on guide to getting everything out of Better Stack's native automation: monitors, escalation policies, webhooks and the Uptime API. Part three covers closing the investigation gap with AI agents.


Why paging got solved and the rest didn't

Detection is a well-defined problem. A check either returns the expected status from enough regions or it doesn't, and an escalation policy either reaches a human within N minutes or moves to the next one. Both can be configured once and trusted.

Everything after the acknowledgment is different in kind:

  • The alert describes a symptom, not a cause. A monitor watches a URL. The cause lives behind it: a pod, a database, a deploy, a certificate. No monitor setting crosses that boundary.
  • The map from monitor to service decays. Which service sits behind api-prod-eu, who owns it and which runbook applies is knowledge that goes stale with every rename and migration. It ends up in two senior engineers' heads.
  • The investigation branches. The next question depends on the last answer. Static escalation rules can route a page, but they can't decide what to check next.
  • The record is a side effect nobody owns. The person who knows the most is the one busiest fixing it, so nobody writes anything down until the next day.

Those four reasons show up as five concrete patterns.

01During

Context hunt

The page says what is down. Which service, which change and which owner is rebuilt by hand.

02Before

Unwired heartbeats

Cron jobs and backups that fail silently because nobody enrolled them.

03During

Logs in another tab

The incident knows the monitor, the logs know the service, only the responder knows the join.

04After

Hand-built timelines

The investigation lived in Slack and scrollback, so the postmortem is reconstructed.

05During

Silent status page

The red bar updates itself. The words customers want come last.

Five places on-call time goes even when detection and paging work.

1. The 3 a.m. context hunt

What it is. The page tells you that something is down, not why. A monitor named api-prod-eu fails its check from three regions. The alert carries the URL, the HTTP status and the response time. It doesn't say which service sits behind that endpoint, which of yesterday's four deploys touched it, whether the database failed over an hour ago, or whether this is the same thing that happened two Tuesdays back. The responder rebuilds all of that by hand, half asleep, before they can form a first hypothesis.

Why it happens. Monitors pile up faster than documentation. A team running 50 to 100 monitors rarely keeps a current map from monitor to service to owner to runbook, because that map goes stale with every rename. So the context hunt is longest exactly when someone less familiar is on call.

How to spot it. Pull your recent incidents from the Uptime API and look at the gap between acknowledged_at and resolved_at:

curl -s "https://uptime.betterstack.com/api/v2/incidents?per_page=50" \
  -H "Authorization: Bearer $BETTERSTACK_API_TOKEN" |
  jq -r '.data[].attributes | [.name, .started_at, .acknowledged_at, .resolved_at] | @tsv'

Acknowledgment is usually fast, under two minutes if your escalation policy works. If resolution routinely lands 30 to 60 minutes later, even for incidents whose eventual fix was one command, the difference is context hunting, not fixing.

Typical cost. 20 to 40 minutes per real incident, concentrated in the responders least familiar with the failing service. On a team with 8 to 12 incidents a month, that's a working day of pure orientation.

2. Heartbeat monitors nobody wired up

What it is. Better Stack's heartbeat monitors are built for things that fail silently: cron jobs, backup scripts, queue consumers, certificate renewals. The job pings a unique URL on every successful run. No ping within the period plus grace time, and you get paged. It's an excellent feature for the jobs that are actually enrolled. The failure mode is the nightly PostgreSQL backup that has been exiting non-zero for three weeks, discovered the day you need it.

Why it happens. A heartbeat needs someone to create the monitor and change the job to ping it. Uptime monitors get created when a service launches, because downtime is loud. Scheduled jobs get written, checked once and forgotten, because their failure is silent by definition. There's no forcing function.

How to spot it. List your heartbeats and compare them against the scheduled jobs you actually run:

curl -s "https://uptime.betterstack.com/api/v2/heartbeats" \
  -H "Authorization: Bearer $BETTERSTACK_API_TOKEN" |
  jq -r '.data[].attributes | [.name, .period, .status] | @tsv'

Then check the other side: crontab -l on a representative host, kubectl get cronjobs -A for Kubernetes, and your CI's scheduled pipelines. Most teams doing this for the first time find they monitor fewer than half of their scheduled jobs.

Typical cost. Zero minutes per incident until it's an incident, and then it's the worst kind: data loss or a compliance gap measured in days or weeks of undetected failure.

3. Log search happens in a different tab than the incident

What it is. Better Stack ships log management (Telemetry) next to Uptime, which should make investigation smooth. In practice the incident page and the log explorer are still two tabs and one human copy-paste apart. The responder reads started_at off the incident, switches tabs, sets the time range by hand, types a query from memory, realizes they filtered the wrong service, and starts over.

Why it happens. The incident knows the monitor, the logs know the service, and only the responder knows the join. Without per-service saved views and consistent structured fields (service, env, request_id), every incident's log search is improvised. A filter like this is quick to run once you know it's the right one:

level:error AND service:checkout-api AND host:prod-eu-*

How to spot it. A ten-minute audit: open your last five incidents and ask whether a saved log view existed for the affected service with the right filters, or whether the responder built the query live. If your Telemetry sources aren't tagged so that a monitor name maps cleanly to a log filter, the answer is already no.

Typical cost. 10 to 20 minutes per incident of query iteration and time-range fiddling. More when logs from the relevant service turn out not to be shipped at all, which is its own 3 a.m. discovery.

4. Incident timelines assembled by hand, after the fact

What it is. The postmortem asks what happened, when, who did what, and what fixed it. Better Stack records detection, acknowledgment, escalations and resolution automatically. The investigation itself happened in Slack threads, terminal scrollback, a Grafana tab and someone's memory. So the next day, an engineer rebuilds the timeline by scraping all four into a doc.

Why it happens. During the incident nobody takes notes, because the person who knows the most is busiest fixing it. The tooling captures the incident's lifecycle, not the investigation.

How to spot it. Compare a recent incident's resolved_at with the timestamp on the finished postmortem, and ask the author how long the writeup took. A 25-minute outage that takes 90 minutes to document is normal. That's also why postmortems quietly stop for "small" incidents, and the same small incident comes back every quarter.

Typical cost. 60 to 120 minutes per documented incident, paid in daytime engineering hours. If skipped, paid later as repeat incidents.

5. Status page updates under pressure

What it is. Better Stack status pages can reflect monitor state automatically, so the red bar shows up on its own. Human-written updates ("we've identified the cause, ETA 20 minutes") don't. Under pressure, the responder's order is: fix it, keep the internal channel updated, then remember the status page. Customers see a red bar with no words on it and route around it. Twenty minutes in, support is fielding duplicate tickets asking whether you know you're down.

Why it happens. A public update needs calm judgment about what to say and what not to promise, exactly when the author has none to spare. So it waits, or falls to whoever feels guilty first.

How to spot it. In Better Stack, open Status pages and compare the last few status reports with the underlying incident timestamps. Measure detection to first update. If it's over 10 minutes, or incidents resolved with no prose update at all, your customers noticed.

Typical cost. 5 to 10 minutes of responder attention per update, which is cheap, plus a support-ticket spike and a trust cost that's harder to measure and larger.


The five gaps on one page

GapHow to spot itTypical cost
Context huntacknowledged_at to resolved_at in the incidents API20 to 40 min per incident
Unwired heartbeatsHeartbeat list vs crontabs, CronJobs and CI schedulesDays of silent failure
Logs in another tabSaved views per service in the last five incidents10 to 20 min per incident
Hand-built timelinesResolution time vs postmortem completion time60 to 120 min per writeup
Silent status pageDetection to first human-written update5 to 10 min, plus support load

The pattern: paging is solved, investigation isn't

Add it up and a "45-minute incident" costs three to four engineer-hours: the context hunt, the log searching, the writeup, the status page. None of that is Better Stack's failure. Uptime monitoring and on-call escalation are the parts of incident response tooling has actually solved. The alert fires within seconds and the escalation policy finds a human within a minute or two, at any hour.

What stays manual is the middle: correlating the alert with what changed, reading the logs against the timeline, forming and testing a hypothesis, keeping the record. Escalation delivers a human quickly and then hands that human all of the work. Part two pushes Better Stack's native features (monitor tuning, escalation policies, webhooks, the Uptime API, log alerts) as far as they go on exactly this problem. It's worth doing, and it has a ceiling.

Shrink the manual middle

The investigation itself can be automated. CloudThinker connects to Better Stack via OAuth in about two minutes, read-only by default. When an incident opens, CloudThinker pulls the failing monitor's context and recent logs, checks the infrastructure behind the endpoint, and posts a likely cause with evidence, so the paged human arrives to a diagnosis instead of a blank timeline. Escalation stays intact, and every change waits for a named approval in Manual mode (Auto Mode explains the levels).

Related reading

Try CloudThinker free with 100 premium credits and no card required, or continue with the hands-on native-tools guide.

betterstackincidentresponseuptimeoncallmonitoringsre