Automating SigNoz Alerts with Native Tools: Rules, Channels, Audits

Open the Alerts page of a SigNoz install that's two years old and count the rules. A team of eight typically finds 60 to 120: rules named test-alert-2, rules pointing at services renamed three quarters ago, four separate rules watching the same latency metric with four different thresholds. Adding a rule takes ninety seconds. Deleting one means knowing it's safe to delete, so nobody does. The result is a SigNoz integration that pages more and means less every sprint.

Part one walked the p99 latency investigation and the signals that matter on an OpenTelemetry-native stack. This part is the hands-on companion: audit the alert rules you already have, rebuild the ones worth keeping with SigNoz's native alerting (query builder, PromQL, ClickHouse queries, evaluation windows, notification channels including webhooks), and know exactly where the DIY ceiling sits. Everything here uses only what ships with SigNoz. Nothing to install.

Step 1: Audit what you have before adding anything

SigNoz exposes its alert rules over the same API the UI uses. Create a personal access token under Settings, API Keys (docs), then pull the full rule inventory:

curl -s -H "SIGNOZ-API-KEY: $SIGNOZ_TOKEN" \
  "https://signoz.your-domain.com/api/v1/rules" \
  | jq '.data.rules | length'

Dump the fields that matter for the audit:

curl -s -H "SIGNOZ-API-KEY: $SIGNOZ_TOKEN" \
  "https://signoz.your-domain.com/api/v1/rules" \
  | jq -r '.data.rules[] | [.id, .alert, .state, .disabled, .createBy, .updateAt] | @tsv' \
  | column -t -s $'\t'

Now sort every rule into one of four buckets:

01Cannot fire

Stale

Targets services or attributes that stopped emitting. Looks like coverage, isn’t.

02Fires, ignored

Noisy

Fires daily, never leads to an action. Bad threshold or a window too short.

03Fires four times

Overlapping

Several rules on one symptom, so one incident becomes several pages.

04Earns its page

Keepers

A user-visible symptom with a threshold someone can defend. Usually 20 to 30%.

Every existing rule lands in one of four buckets. Disable the first three for two weeks before deleting.

Stale. Rules that reference services or attributes that no longer emit data. Their state sits at inactive permanently, not because things are healthy but because the query returns nothing. Cross-check each rule's target against Services (the service list only shows what's actually reporting). A rule that can't fire is worse than no rule, because it looks like coverage.

Noisy. Open a suspect rule and check its history (Alerts, click the rule, Alert History), which shows firing frequency and duration over the past weeks. Anything firing daily that never produced a human action is noise. The usual causes: a threshold set from a bad week ("alert at 500ms because we saw 480ms once"), or an evaluation window so short that normal variance trips it.

Overlapping. Sort your dump by the metric or service each rule targets. Four rules on checkout latency at 400ms, 500ms, 800ms and 1s means one incident produces four pages. Keep two at most: a warning-severity rule at the level someone should look, and a critical-severity rule at the level someone should act.

Keepers. Rules mapped to a symptom a user would notice (latency, error rate, saturation about to become latency), with a threshold someone can defend. These are usually 20 to 30% of the total.

Disable, don't delete, the first three buckets for two weeks. If nothing was missed, delete them. The audit takes two to three hours and typically cuts page volume by 40 to 60% before you've tuned anything.

Step 2: Rebuild rules with the right query type

SigNoz alert rules run against any signal (metrics, traces, logs, exceptions), and each rule is built from one of three query flavors (docs). Choosing the right one is most of the work.

01Default

Query builder

Metric, aggregation, filter, threshold. Right for most rules.

02Porting

PromQL

Existing Prometheus rules, absent() and predict_linear().

03Raw data

ClickHouse SQL

Per-endpoint percentiles from raw spans and logs. Always bound the time range.

Three ways to write the same rule. Pick the simplest one that can express the condition.

Query builder: the default

Alerts, New Alert Rule, Metrics-based Alert gets you the visual builder: pick a metric, aggregation, group-by and threshold. For the standard span metrics SigNoz generates from your traces, a p99 latency rule on one service looks like this: metric signoz_latency, aggregation p99, filter service_name = checkout, threshold above 800 ms.

The two settings engineers skip and shouldn't:

Evaluation window (evalWindow): how much data each evaluation considers. Five minutes is a sane default for latency. One minute makes every GC pause a page.
Match condition: at least once, all the time, on average or in total within the window. For p99 latency, on average over 5 minutes pages on sustained degradation; at least once pages on a single bad interval. Most noisy latency rules from your audit are "at least once with a 1-minute window." Rebuild them as "on average over 5 minutes" and the flapping stops.

PromQL: when you already know the expression

The same rule in PromQL, for teams migrating from a Prometheus stack:

histogram_quantile(0.99,
  sum(rate(signoz_latency_bucket{service_name="checkout"}[5m])) by (le)
) > 800

The signoz_latency histogram is recorded in milliseconds, so the threshold stays in ms, the same 800 you set in the builder.

PromQL rules are the right choice when you're porting existing Prometheus alert rules verbatim, or when you need functions the builder doesn't expose: predict_linear for disk-full forecasting, or absent() for missing-data alerts. absent() is also how you make the "stale rule" failure mode from your audit page you instead of staying silent.

ClickHouse queries: alerts on raw trace and log data

Because SigNoz stores everything in ClickHouse, you can alert on queries the metric pipeline can't express, for example p99 computed from raw spans for one endpoint only:

SELECT
  toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval,
  quantile(0.99)(durationNano) / 1e6 AS p99_ms
FROM signoz_traces.distributed_signoz_index_v2
WHERE serviceName = 'checkout'
  AND name = 'POST /api/pay'
  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY interval;

The rule engine fills in the start_datetime and end_datetime template variables at evaluation time (docs). Note the table name: installs older than roughly v0.64 use signoz_index_v2 as above; newer versions moved to signoz_index_v3 with a different column layout, so run SHOW TABLES FROM signoz_traces on your cluster before copying.

This power comes with a cost warning. A ClickHouse alert query runs at every evaluation interval, so an unbounded scan over a week of spans will hurt the very cluster you're monitoring. Always keep the timestamp bounds.

Log-based alerts follow the same pattern against signoz_logs.distributed_logs. The classic use case is paging when severity_text = 'ERROR' counts for one service jump above baseline.

Step 3: Notification channels, and what webhooks actually give you

Channels live under Settings, Alert Channels (docs): Slack, PagerDuty, Opsgenie, email, Microsoft Teams, and the one that matters for automation, webhook.

SigNoz's alert delivery is Alertmanager-compatible, so a webhook channel POSTs the standard payload to your endpoint:

{
  "receiver": "ops-webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "checkout-p99-critical",
        "severity": "critical",
        "service_name": "checkout",
        "ruleId": "42"
      },
      "annotations": {
        "description": "P99 latency on checkout above 800ms for 5m",
        "summary": "checkout latency breach"
      },
      "startsAt": "2026-07-13T04:12:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "c1f9a2b8e77d4c01"
    }
  ],
  "groupLabels": { "alertname": "checkout-p99-critical" },
  "version": "4"
}

Everything your receiving automation needs is here: fingerprint for deduplication, startsAt to anchor the incident window, and labels to route on. Typical DIY builds on top of this payload: a small service that opens a Jira ticket per fingerprint, a Lambda that posts the alert plus a pre-built SigNoz trace-search link into the incident channel, or a bridge into a runbook automation tool. Use the Test button on the channel before trusting it, because a webhook receiver that returns 500 fails silently otherwise.

The pre-built trace link is the most useful of those, and it shows the limit clearly. A receiver can compute the window from startsAt and hand the on-call a filtered view:

# alert is one entry from payload["alerts"]; trace_explorer_url and slack are your own helpers
start = parse(alert["startsAt"])
window = (start - timedelta(minutes=15), start + timedelta(minutes=5))
link = trace_explorer_url(service=alert["labels"]["service_name"],
                          start=window[0], end=window[1],
                          min_duration_ms=800)
slack.post(channel="#inc-checkout", text=f"{alert['annotations']['summary']}\n{link}")

That saves the on-call two clicks. It doesn't open the traces, compare them against yesterday, or notice the deploy four minutes earlier.

A routing convention that keeps channels sane: label rules with severity, send critical to the pager and warning to Slack only. One rule, one severity, one destination.

Step 4: Dashboards worth keeping

The same audit logic applies to dashboards. Most instances pile up dozens, and four earn their keep:

Per-service golden signals. Rate, error % and p50/p99 duration from the built-in Services view. You get this free, so don't rebuild it.
Alert-adjacent dashboards. One per critical alert rule, showing the alerting query plus its likely causes (a latency alert gets a dashboard with latency, error rate, downstream call duration and host saturation). Link it in the rule's annotation so the page carries its own context.
Deploy correlation. Your key latency panels with deployment markers, since "what changed?" is the first triage question.
ClickHouse self-monitoring. Disk, ingest rate and query latency of the SigNoz cluster itself. A self-hosted observability stack that falls over during an incident is a story you only want to live once.

Delete or archive the rest. A dashboard nobody opened in 90 days is UI debt.

Where SigNoz sits next to Datadog (the honest version)

The SigNoz vs Datadog question comes up in every one of these audits, and it deserves a neutral answer. SigNoz gives you an OTel-native pipeline, your data in your own ClickHouse, and a cost model that scales with your hardware rather than per-host or per-GB list pricing. In return, you operate the cluster, and the alerting features above are yours to configure and maintain. Datadog gives you managed polish, a larger integration catalog and mature correlation features out of the box, at a per-unit price mid-market teams feel at scale, and with your telemetry living in someone else's system. Neither is wrong. The trade is control and cost predictability versus outsourcing the operations. Everything in this guide is the price of the control side.

The ceiling: channels deliver alerts, they don't read traces

Run the audit, rebuild the rules, wire the webhooks, and you have a good alerting setup. Be honest about what you don't have:

01Audit

Point in time

Rules regrow after every retro. Nobody schedules the pruning.

02Thresholds

Decay

Right for last quarter’s traffic, wrong after the next architecture change.

03Webhooks

Move, not read

They open tickets and post links. They don’t open the trace.

Where the native setup stops. The teal cell is the one that still runs on a person at 3 a.m.

The audit is point-in-time. Rules regrow. Every incident retro spawns a new rule, and nobody schedules the pruning. The 90-rule pile you just cleaned is where you'll be again in 12 to 18 months unless auditing becomes a calendar event.

Thresholds decay. The 800ms threshold that was right at last quarter's traffic is wrong after the next architecture change. Static rules don't renegotiate themselves.

Webhooks move alerts; they don't investigate them. The payload above can open a ticket and post a link. It can't open the trace, compare the incident window against baseline, notice that p99 broke four minutes after a deploy, or read the ClickHouse saturation panel. When the page arrives at 3 a.m., the query builder, the trace waterfall and the correlation work are still a person's job: the exact workflow from part one, run by whoever is on call, every time.

The reason is the same in all three. Every native feature is deterministic. It does the same thing for every alert that matches. Investigation isn't: what you query next depends on what the last query returned, and no rule or webhook script can hold that tree in advance.

What comes next

The investigation your webhook can't do can be automated. CloudThinker connects to SigNoz read-only, picks up the firing alert, queries the traces, metrics and logs around the incident window the way an engineer would, correlates with deploys and cloud state, and proposes remediation. It acts within Auto Mode, so you decide what runs on its own and what waits for approval. That's part three of this series.

Or see it against your own stack now: Try CloudThinker free with 100 premium credits and no card required.