SigNoz Integration with AI Agents: From Firing Alert to Named Cause

Think about your last incident. Of the total time from alert to fix, how much went to finding out what happened? For most teams it's most of it, and none of that was SigNoz's fault. The spans were in ClickHouse, the rule fired, the webhook delivered. What the stack couldn't supply was the next step: someone to open the traces, compare the window against baseline, and connect the spike to the deploy that caused it.

That's the last gap in a SigNoz integration, and this final part of the series closes it. Part one covered the signals that matter on an OpenTelemetry-native stack and walked the p99 investigation: service overview, trace filtering, span drill-down. Part two built everything SigNoz's native tooling offers (rule audits, query-builder and ClickHouse alert rules, notification channels, webhook payloads) and ended at a hard ceiling: channels deliver SigNoz alerts, they don't read traces.

This part covers the layer that does. CloudThinker connects to your SigNoz instance read-only, picks up firing alerts, runs the part-one investigation the way an engineer would, correlates with what SigNoz can't see (deploys, cloud state, scaling events), and names the likely cause with evidence attached. It acts only when you've allowed it to. It isn't a SigNoz replacement. Your OTel pipeline, dashboards and alert rules stay where they are. CloudThinker is the action layer on top.

Why webhooks stop where investigation starts

Part two's webhook receiver does the same thing for every payload: compute a window from startsAt, build a trace link, post it. That's correct for delivery, and it's why it can't investigate. Investigation branches. If the slow spans are all on the new version, you read the deploy. If they're on every version, the deploy is irrelevant and you look at the database. If one client span dominates, you look upstream. Nobody can write that tree down in advance, so it stays in the head of whoever is on call.

Connecting SigNoz: one API key, about five minutes

01Observability

SigNoz

Alert rules and stateRules API, alert history
Traces, metrics, logsOTel data in ClickHouse
ServicesRED metrics per service

Read-only API key

02Investigation

CloudThinker

Pick up the alertRule, query, firing history
Query the windowSlow spans, logs, baselines
Correlate changesRollouts, scaling, failovers

No collector changes

03Where the team works

Chat and on-call

Slack or TeamsFinding and approvals in the thread
PagerDutySame escalation, evidence attached
OpsgenieSame escalation, evidence attached

Nothing new to watch

SigNoz keeps collecting and alerting. CloudThinker reads the same API your part-two curl commands hit, then looks where SigNoz can't.

The connection is the same one you used for the part-two rule audit: a SigNoz API key with read access to the query and alert APIs. Create it under Settings, API Keys in your SigNoz instance (docs), paste the key and your instance URL into CloudThinker, and the first scan starts. For self-hosted installs the only infrastructure question is reachability: CloudThinker needs HTTPS access to the same API endpoint your own curl commands from part two hit.

No collector changes, no new instrumentation, no write access at connection time. The SigNoz connection guide has the step-by-step. If you also connect your cloud accounts (AWS, GCP, Azure, Kubernetes, all read-only), the agent can follow an incident across the boundary where SigNoz's view ends. Most causes live on the other side of it.

One alert, end to end

01Signal

Rule fires

A rule moves to firing. Its query, threshold and history come along.

02Readings

Hypotheses

Deploy regression, shared saturation or a slow upstream.

03Evidence

Query each

Spans by version, host metrics, client spans, rollout history.

04Output

Named cause

What broke, the traces that show it, a fix with a rollback note.

Change correlation is the check people run last. The agent runs it in the first minute.

Here's a single alert followed all the way through. The details are illustrative; the method is the point. At 04:12 the checkout-p99-critical rule from part two fires: p99 on checkout above 800ms for five minutes. The rule has fired twice in 30 days, both real, so this isn't the flapping kind.

1. Name the readings. Three explanations fit, and each predicts something different.

Deploy regression. Something shipped recently changed a hot path. Predicts: the slow spans carry the new service.version, and pods still on the old version stay fast.
Shared saturation. The database or node under checkout is overloaded. Predicts: other services on the same database slow down too, across all versions.
Slow upstream. The payment provider degraded. Predicts: one client span to the provider holds most of the time, and p50 barely moves.

2. Query the traces, split by version. The same Traces explorer query from part one, with a group-by that only works if you set service.version in OTEL_RESOURCE_ATTRIBUTES:

Traces explorer, 04:00 to 04:30
  filter:    service.name = checkout AND duration_nano >= 800000000
  aggregate: p99(duration_nano), count()
  group by:  service.version, name

Back comes a clear split. On v2.41, the SELECT orders span has a p99 of 1.9s and appears in nearly every slow trace. On v2.40 (two pods that drained until 04:10), the same span sits at 40ms. The provider's client span is flat at about 180ms on both versions. The upstream reading drops out.

3. Check the shared-saturation reading. If the database were the problem, its other callers would feel it. A PromQL check on the span-derived latency metric:

histogram_quantile(0.99,
  sum(rate(signoz_latency_bucket{service_name=~"orders|billing"}[5m])) by (le, service_name)
)

Both services are flat through the window, and the database host's CPU is at 35%. Shared saturation drops out too.

4. Read the change. The agent checks the rollout history for the window:

kubectl rollout history deployment/checkout -n prod
kubectl get events -n prod --field-selector involvedObject.name=checkout --sort-by=.lastTimestamp

checkout v2.41 rolled out at 04:08. The db.statement attribute on the slow span shows the query now filters on orders.status, a column the previous version didn't touch and that has no index. The query planner went from an index scan to a sequential scan on a large table.

5. Hand back a finding, not a forward. What lands in the incident channel:

checkout-p99-critical fired at 04:12. Cause: checkout v2.41 (rolled out 04:08) added a filter on orders.status, which has no index; the SELECT orders span went from 40ms to 1.9s p99. Not the database (orders and billing flat, DB CPU 35%), not the payment provider (client span flat). Three example traces attached. Options: roll back to v2.40 now (fast, reversible), or add an index on orders.status through a reviewed migration. Proposed: rollback, then a ticket for the index. Waiting for approval.

The on-call reads one message instead of opening six tabs. The investigation took about three minutes, and every query is in the thread for anyone to re-run.

The agent does what you've allowed, nothing more

Disabling a rule the audit already proved dead and rolling back production aren't the same kind of decision. CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. Escalation stays intact either way. If the evidence says real outage, the agent pages through your existing PagerDuty or Opsgenie path exactly as before, with the investigation already attached. Every step lands in an audit trail: what fired, what was queried, what was concluded, what was proposed, who approved and what changed.

What the first pass typically finds

Before the first live incident, connecting runs a hygiene pass over your alert configuration: the part-two audit, done in minutes instead of an afternoon. The numbers below are illustrative, a composite of what a first scan tends to surface on a self-hosted instance with 20 to 40 services and a rule pile that's two years old. Yours will differ.

Finding	Detail	Impact
Stale rules	24 rules target services that no longer emit data	Coverage that cannot fire
Flapping rules	12 rules fired 10+ times in 7 days with no human action taken	~half of total page volume
Overlapping latency rules	4 rules on the same service at 400ms/500ms/800ms/1s	Four pages per incident
1-minute evaluation windows	19 latency rules page on single-interval spikes	GC pauses become pages
Unbounded ClickHouse queries	2 alert queries scan a week of spans at every evaluation	Load on the cluster you monitor
Services with no alert at all	3 production services in the Services tab, zero rules attached	Incidents found by customers
Missing error-log rules	ERROR-volume step changes precede symptoms; nothing watches them	Minutes of lost lead time

Look at the shape. Almost none of this is "add more monitoring." It's your existing SigNoz alerts being wrong in specific, fixable ways, and each row arrives with a proposed fix that waits for approval rather than a to-do item.

Why this beats the traditional workflow

	Webhooks and manual triage	With CloudThinker on top
At 04:12	A Slack message and a trace link	A finding with the cause named
Deploy correlation	Checked last, by hand	Checked first, by version
Ruling things out	Whatever the on-call remembers to check	Each reading tested and written down
Outside SigNoz	Separate consoles, separate logins	Rollouts, scaling and infra in the same pass
Rule hygiene	A yearly audit, then regrowth	Continuous, with proposed fixes
Actions	Whatever the script does	Gated by Auto Mode, audited

The deeper difference is where the investigation lives. With webhooks, it lives in a person's head at 3 a.m. With an agent on top, it lives in the thread: every reading, every query and every result, so the next on-call can check it and the team can tell which rules and dashboards actually helped.

Prompts to try in your first session

CloudThinker is conversational. You ask in plain language, and the answer cites the underlying SigNoz data so you can verify it rather than trust it:

"The checkout p99 rule just fired. Pull the traces over 2 seconds in the incident window, break down the worst spans by service version, compare against the same window yesterday, and check for deploys or scaling events since 04:00. Propose only, change nothing."

"Audit our SigNoz alert rules: which are stale, which flapped more than five times last week, and which evaluation-window or threshold changes would cut the noise without losing real signal?"

"For the payment-service error spike this morning, correlate the error logs with traces and tell me whether the cause is inside our stack or the upstream provider. Show me the spans either way."

Run the first one during your next real incident. Compare the agent's three-minute finding with what your on-call assembled by hand, and you'll know whether it earns a place in your escalation path.

What the agent won't do

Connecting anything with "AI" in the name to production observability should raise questions. Here are the answers:

Read-only by default. The API key grants query and read access. Acting on your systems (silencing a rule, restarting a workload, rolling back a deploy) needs separately granted write access and still goes through Auto Mode.
No silent changes. Every change shows up in chat with its evidence, whether it ran on its own or waited for approval.
No bypassed escalation. The agent never decides an incident isn't worth waking someone for. Your paging rules stay yours, explicit and logged.
No unauditable steps. Every query, conclusion, proposal, approval and change is in the trail with its evidence.

Since the SigNoz vs Datadog question ran through this whole series: the agent layer doesn't tilt that decision. SigNoz gives you OTel-native portability, self-host control and a cost model tied to your hardware; Datadog gives you managed polish and breadth at per-unit prices. Both deliver alerts, and neither investigates them. The gap this series is about exists on either stack, which is why it gets its own layer rather than another dashboard.

Closing the loop

Part one showed that on a well-instrumented SigNoz stack the evidence is always three clicks away, and that a person still has to make the clicks. Part two automated everything deliverable and stopped at the ceiling: webhooks move alerts, they don't read traces. The distance that remains, from alert fired to cause named with evidence, is what an agent layer closes. Teams that add investigation on top of an existing SigNoz setup typically cut time to diagnosis from tens of minutes to under five, and clean up a couple of years of alert-rule debt in the first week along the way.