How To

How to Automate Incident Response with Better Stack's Native Tools

Get the most out of your Better Stack integration with native features alone: uptime monitors tuned with multi-region checks and confirmation periods, heartbeats for cron jobs and queue workers, escalation policies built around on-call calendars, incident automation via the Uptime API and outgoing webhooks, log search and alerting in Better Stack Telemetry, and status pages that update themselves. Part two of our Better Stack incident response series — exact settings, copy-pasteable curl examples, and an honest look at the ceiling: escalation finds a human fast, but the investigation is still manual.

·
betterstackincidentresponseuptimeoncallmonitoringsre
Cover Image for How to Automate Incident Response with Better Stack's Native Tools

How to Automate Incident Response with Better Stack's Native Tools

Your escalation policy already works. A monitor fails, the incident opens, the on-call engineer's phone rings inside two minutes, and they acknowledge before the second escalation step ever fires. Your Better Stack integration is doing exactly what it was configured to do — and then the automation stops, and the engineer starts the same manual investigation they ran last month.

This article is about pushing that boundary as far as it will go with native features alone. In part one of this series we mapped where on-call time actually leaks: the 3 a.m. context hunt, cron jobs nobody wired to a heartbeat, timelines rebuilt by hand, log search living in a separate tab, status pages updated last. Here we work feature by feature — uptime monitors, heartbeats, escalation policies, the Uptime API and incident webhooks, Logs, and status pages — with exact settings and copy-pasteable calls. Budget an afternoon; the payoff is fewer false pages and much richer context when a real one lands.

All API examples use the Uptime API v2 with a token from Uptime → Integrations → API tokens (API docs):

export BETTERSTACK_TOKEN="your-api-token"

1. Uptime monitors: frequency, regions, confirmation period

Everything downstream — escalations, webhooks, status pages — keys off monitors, so tune these first. Four settings do most of the work:

  • check_frequency — 30 seconds on paid plans (the free tier checks every 3 minutes). For anything customer-facing, 30s is the difference between catching a deploy gone bad and hearing about it on Twitter.
  • regions — check from all four (us, eu, as, au). Better Stack re-verifies a failure from a second location before opening an incident, so multi-region checks convert "one PoP had a bad route" from a page into a non-event.
  • confirmation_period — how long the check must stay failing before anyone is alerted. 60–120 seconds absorbs restarts and blips; 0 means every flap pages someone.
  • recovery_period — how long the check must stay green before the incident auto-resolves. Keep it at 180s or a flapping service will "resolve" itself mid-outage.

Create one via the API:

curl -X POST https://uptime.betterstack.com/api/v2/monitors \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "monitor_type": "expected_status_code",
    "url": "https://api.example.com/health",
    "expected_status_codes": [200],
    "check_frequency": 30,
    "regions": ["us", "eu", "as", "au"],
    "confirmation_period": 120,
    "recovery_period": 180,
    "request_timeout": 15
  }'

Prefer expected_status_code or keyword monitors over the plain status type for health endpoints — a 302 to your login page is "up" to a status check and very much down to your users.

Maintenance windows are the fix for the deploy-night page. Set them per monitor (Monitor → Configure → Maintenance window, or maintenance_days, maintenance_from, maintenance_to, and maintenance_timezone in the same API payload) so the Tuesday 01:00–02:00 UTC deploy never opens an incident. If your team silences monitors by pausing them instead, you have a standing incident risk: paused monitors don't come back on their own.

2. Heartbeats: cover the things that don't listen on a port

Uptime monitors probe from outside; heartbeats invert the direction for cron jobs, queue workers, and backups — the job must check in, or Better Stack opens an incident. Part one found these are the most commonly unwired class of failure; wiring one takes two calls.

Create the heartbeat (period = expected interval, grace = slack before alerting, both in seconds):

curl -X POST https://uptime.betterstack.com/api/v2/heartbeats \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "nightly-db-backup", "period": 86400, "grace": 3600}'

The response includes a unique heartbeat URL. Append it to the job itself so the ping only fires on success:

0 2 * * * pg_dump mydb | gzip > /backups/mydb.sql.gz \
  && curl -fsS https://uptime.betterstack.com/api/v1/heartbeat/YOUR_TOKEN

You can also report an explicit failure immediately — without waiting out the grace period — by hitting the same URL with /fail appended. Inventory rule: every cron job and queue consumer that matters gets a heartbeat, and the heartbeat call lives in the job's own definition, not in a wiki.

3. Escalation policies and on-call calendars

Set up the on-call calendar first (Uptime → On-call & escalations): a rotation with explicit handoff times, plus overrides for holidays. Then build escalation policies that reference whoever is on-call rather than named humans — named humans go on vacation.

A policy shape that holds up in practice:

  1. Immediately: push + phone call to the current on-call person. Calls wake people; push notifications alone do not survive do-not-disturb misconfiguration.
  2. After 5 minutes unacknowledged: escalate to the entire team.
  3. After 15 minutes: escalate to the engineering lead.

Attach different policies to different monitor groups — checkout API failures deserve the phone call; the staging environment deserves an email. Audit what you actually have configured:

curl "https://uptime.betterstack.com/api/v2/policies" \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN"

If that returns one default policy attached to everything, severity routing is your highest-leverage fix in this whole article — undifferentiated paging is how teams train themselves to ignore pages.

4. The incidents API and incident webhooks

Better Stack records every incident with cause, timestamps, and acknowledgment state. That data is scriptable. Pull last week's incidents for a review meeting:

curl "https://uptime.betterstack.com/api/v2/incidents?from=2026-07-06&to=2026-07-13" \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN"

Acknowledge or resolve programmatically — useful when your deploy tooling knows a rollback just fixed the problem:

curl -X POST "https://uptime.betterstack.com/api/v2/incidents/INCIDENT_ID/resolve" \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"resolved_by": "deploy-bot@example.com"}'

Outgoing webhooks (Uptime → Integrations) are the extension point: Better Stack POSTs JSON to your endpoint when an incident starts, is acknowledged, or resolves. Teams typically use this to open a ticket automatically, post an enriched message to a Slack channel with runbook links, or trigger a first-response script — restart the service, capture a thread dump — before a human is even awake. This is real automation, with one caveat we'll return to: you write, host, and maintain every one of those receivers yourself.

5. Better Stack Logs: search and alert next to your incidents

The "logs in a separate tab" problem from part one has a partial native fix: ship logs into Better Stack Telemetry so the incident and the evidence live in one product (Logs docs). Create a source (Telemetry → Sources → Connect source), then ship via Vector, the OpenTelemetry collector, or plain HTTP with the source token:

curl -X POST https://in.logs.betterstack.com \
  -H "Authorization: Bearer $SOURCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "checkout failed: connection pool exhausted", "level": "error", "service": "api"}'

Two features earn their keep during incidents. Live tail with filtering gives you grep-speed search across services while the incident is open. Log-based alerts let you page on what monitors can't see — a spike of 5xx responses in your access logs, or any occurrence of connection pool exhausted — and route through the same escalation policies as your uptime checks. An HTTP monitor confirms the symptom; a log alert often names the region of the cause, which is a meaningfully better page to wake up to.

6. Status pages that update themselves

Wire your status page directly to monitors so the "investigating" state appears without a human touching anything mid-incident:

# Create the page
curl -X POST https://uptime.betterstack.com/api/v2/status-pages \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"company_name": "Example", "subdomain": "status-example", "timezone": "UTC"}'

# Attach a monitor as a public resource
curl -X POST https://uptime.betterstack.com/api/v2/status-pages/PAGE_ID/resources \
  -H "Authorization: Bearer $BETTERSTACK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"resource_id": MONITOR_ID, "resource_type": "Monitor", "public_name": "API"}'

When a linked monitor's incident opens, the component flips automatically; when it recovers, it flips back. Human-written updates are then reserved for what humans are actually needed for — explaining impact and ETA — instead of the mechanical "we are aware" post.

The ceiling: escalation finds a human, not a cause

Configure everything above and you have a genuinely good setup: false pages mostly gone, silent cron failures covered, severity-routed escalation, logs one click from the incident, a status page that keeps itself honest. Be equally clear about what you do not have:

Detection stops at the boundary Better Stack can see. It observes the symptom — the 500, the timeout, the missed heartbeat. The cause lives on the other side: the disk that filled, the pod the OOM killer took, the connection pool a bad deploy exhausted. No monitor setting reaches there.

Investigation is manual, every single time. The escalation policy's job ends when a human acknowledges. Which monitor, which service, what changed in the last hour, which log line matters — that 30–40 minute sequence from part one is untouched by everything configured here.

Webhook automation is code you own. Every receiver is a script your team writes, hosts, secures, and updates when the payload or the infrastructure changes. Most teams build two, then stop.

Native tooling optimizes time-to-human. The larger cost — time-to-cause — still comes out of an engineer's night.

What comes next

The investigation itself is what part three of this series automates: CloudThinker agents connect to Better Stack via OAuth in about two minutes, read-only by default, pick up incidents as they open, pull the failing monitor's context and recent logs, and investigate the infrastructure behind the symptom — so the on-call engineer arrives to a diagnosis instead of a blank timeline, with the escalation chain fully intact. See what it finds in your own stack: Try CloudThinker free — 100 premium credits, no card required.