SigNoz Integration: The Observability Signals That Actually Matter

At 14:07 the p99 latency on your checkout API jumps from 240ms to 2.1s. Not every endpoint, just one, on one service. The trace that explains why is already sitting in SigNoz, three clicks away. It stays unopened for forty minutes, because the person on call is scrolling a Slack channel asking "anyone else seeing slowness?"

That gap, between data that exists and a person who has read it, is where most triage time goes. A good SigNoz integration closes part of it. It puts traces, metrics and logs from every service into one queryable place, on an open standard, at a cost model a mid-market team can sustain.

This guide covers what that integration looks like, which signals deserve your attention, how a p99 investigation flows through the UI, how alert rules work across signal types, and where the gap remains even after everything is wired up.

Part two is a hands-on guide to alert automation with SigNoz's native features. Part three covers putting an AI action layer on top of your SigNoz alerts.

How SigNoz sees your stack

SigNoz is OpenTelemetry-native end to end. There's no proprietary agent. Your services emit traces, metrics and logs through standard OTel SDKs and the OTel Collector, and SigNoz stores all three signal types in ClickHouse, a columnar database built for high-cardinality, time-ordered data. One store, three signals, one query surface.

The practical consequence: instrumenting a service for SigNoz is instrumenting it for OpenTelemetry. For most languages you don't touch application code. You set environment variables and let auto-instrumentation do the work:

export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<signoz-otel-collector>:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=$GIT_SHA

Point that at your SigNoz OTel Collector (port 4318 for OTLP over HTTP, 4317 for gRPC) and the service shows up in the Services tab within minutes. The instrumentation docs cover the per-language specifics. Java and Python are near zero-code; Go needs explicit SDK wiring.

Note the service.version attribute. It's optional, and most teams skip it. Set it anyway. It stamps every span with the release that produced it, which turns "did the deploy cause this?" from a timeline comparison into a filter. Part three leans on it heavily.

Because it's all OTel, nothing you do here locks you in. The same instrumentation exports to any OTLP-compatible backend by changing one endpoint. That matters more than it sounds, as the sidebar below explains.

The signals that matter for a mid-market stack

You don't need hundreds of dashboards. For a stack of 10 to 50 services, a short list of signals catches most real incidents:

01Services

RED metrics

Rate, errors and duration per service, derived from traces for free.

02Tail

p99 latency

The 1% of requests that time out. Averages hide them.

03Endpoints

Error rate by route

A healthy-looking service can hide one route failing at 40%.

04Dependencies

Client spans

Database, cache and third-party calls. The "us or them?" answer.

05Infra

Saturation

CPU throttling, memory pressure, and disk on ClickHouse itself.

06Logs

ERROR volume

A step change often shows up minutes before the user-visible symptom.

Six signals that catch most real incidents on a 10 to 50 service stack.

Per-service RED metrics. Request rate, error rate and duration (p50, p90, p99) for every service. SigNoz derives these from trace data; the Services tab is this list, precomputed.
p99 latency, not average. Averages hide the 1% of requests that time out and page you at 2 a.m. A p99 that doubles while p50 holds steady is the classic signature of one slow dependency, one bad shard or one pathological query.
Error rate by endpoint, not just by service. A service at 0.4% errors looks healthy until you see that one route is at 40% and the rest are clean.
External call latency. Database, cache and third-party APIs show up as client spans. When your p99 spikes, the first question is almost always "us or a dependency?"
Saturation from infra metrics. CPU throttling, memory pressure, and disk on your ClickHouse host itself. Ship these through the OTel Collector's hostmetrics receiver or the Kubernetes infra integration.
Error-log volume by service. A step change in ERROR-level log volume often comes minutes before the user-visible symptom.

One useful check on day one: open Services and confirm every production service you expect is there. A service missing from that list is a service whose incident you'll debug blind.

Anatomy of a p99 investigation

01Scope

Service overview

p99 alone or p50 too? Did traffic change at the same moment?

02Filter

Slow traces

One service, one route, a duration floor, sorted worst first.

03Drill

Span breakdown

One span usually holds most of the time. That span is the lead.

04Correlate

What changed

Deploys, scaling, failovers. The step SigNoz leaves to you.

The p99 investigation SigNoz is built around. The first three steps take minutes once you know them; the fourth happens outside the tool.

Here's the workflow SigNoz is built around. Practice it before an incident, not during one. Start the same way an experienced on-call does, by naming what the spike could be:

Overload. Traffic jumped and everything slowed. Predicts: request rate up, p50 and p99 both rising, across endpoints.
One slow dependency. A database query, cache or third-party call degraded. Predicts: p99 up, p50 flat, one client span dominating the slow traces.
One bad instance. A noisy neighbor or a throttled pod. Predicts: slow traces concentrated on one host.name or k8s.pod.name.

Each step below tests those readings.

Step 1: Service overview. Open Services, then checkout-api. The overview shows latency percentiles, request rate, error rate and Apdex over time. Is the spike in p99 only, or p50 too? Did the request rate change at the same moment? A p99 spike with flat p50 and flat traffic rules out overload and points at a subset of requests.

Step 2: Trace filtering. From the latency chart, drill into traces for the affected window. In the Traces explorer, filter to the slow subset: service name, the specific route, and a duration floor so you only see the slow ones:

service.name = checkout-api
http.route = /api/v1/checkout
duration_nano >= 2000000000

Attribute names vary slightly between SigNoz versions (older trace explorers expose durationNano), so pick from the filter's autocomplete. Sort by duration, descending. You now have the ten worst requests of the incident in front of you, with full context. Group the same filter by host.name and you've tested the bad-instance reading in one more click.

Step 3: Span drill-down. Open one of those traces. The flame graph shows the full request tree with per-span timings. In practice the answer is visible within seconds: one span holding 1.9 of the 2.1 seconds, such as a SELECT against the orders table, a call to a payment provider, or a lock wait. The span's attributes carry the query text, the peer address and the status code. That's your lead.

For someone who knows the workflow, this takes three to five minutes. For someone who starts in Slack, it takes much longer. The tool is fast. The bottleneck is a person deciding to look, knowing where to look, and being awake enough to read what they see. And the fourth step, connecting that slow span to whatever changed (a deploy, a migration, an autoscaling event), happens outside SigNoz entirely.

Alert rules across signal types

SigNoz supports alert rules on every signal it stores. You don't export logs to a second system just to alert on them.

01Metrics

Thresholds

Query builder or PromQL over metrics and span-derived histograms.

02Traces

Span counts

Errored or slow spans on a route over a window.

03Logs

Log volume

ERROR-level counts per service, in the builder or ClickHouse SQL.

04Exceptions

New groups

New or recurring exception groups from the Exceptions tab.

05Anomaly

Baselines

Deviation from a learned seasonal pattern instead of a fixed line.

Every signal SigNoz stores can carry an alert rule. No second system just to alert on logs.

Metric-based alerts, built with the query builder or raw PromQL. The canonical p99 alert:

histogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="checkout-api"}[5m])) by (le)) > 2000

Trace-based alerts: thresholds on trace-derived data, for example the count of spans with has_error = true on a given route over the last five minutes.
Log-based alerts: for example, ERROR-level log count per service above a threshold, built in the logs query builder or as a ClickHouse query.
Exception-based alerts: fire on new or recurring exception groups from the Exceptions tab.
Anomaly-based alerts: deviation from a learned seasonal baseline instead of a static threshold, useful for metrics with strong daily cycles.

Two configuration habits pay for themselves. First, set evaluation windows deliberately. A p99 alert evaluated over one minute pages you for every garbage-collection pause; a five-minute window with a sustained-breach condition cuts the noise sharply. Second, route by severity from day one: pages to PagerDuty or Opsgenie, warnings to Slack, everything else to a channel nobody is required to read. Part two walks through the full alert-rule setup, webhook payloads, and how to audit stale rules.

You'll need that audit, because alert rules rot. Thresholds set during an incident stay forever, services get renamed and their rules silently match nothing, and two teams independently alert on the same symptom. A quarterly pass through Alerts, sorted by last-fired time, finds rules that have never fired (probably broken) or fire daily and get ignored (definitely broken).

Sidebar: where SigNoz sits next to Datadog

The "SigNoz vs Datadog" question comes up in almost every evaluation, and it deserves a plain answer rather than a cheer for either side. They're different trade-offs.

Cost model. Datadog prices per host plus per product (APM, logs and custom metrics are each metered separately), which is predictable per unit but compounds as you adopt more of the platform. SigNoz Cloud prices on usage (per GB ingested, per metric sample), and self-hosted SigNoz costs whatever your ClickHouse infrastructure costs. For log-heavy and trace-heavy mid-market workloads the usage-based or self-hosted math often comes out substantially lower, but model your volumes rather than trusting anyone's blog.
Self-host control. Self-hosted SigNoz keeps telemetry in your VPC, which matters for data-residency and compliance constraints, and puts retention entirely in your hands. The cost of that control is real: you operate ClickHouse, you plan capacity, you do the upgrades.
OTel standardization versus integrated polish. SigNoz bets everything on OpenTelemetry, which means portable instrumentation and no agent lock-in. Datadog supports OTel too, but its strength is the opposite bet: hundreds of turnkey integrations, mature ML-driven features, and the polish of a large managed platform. Teams that value breadth and hands-off operation reasonably choose it.

If your constraints are budget, OTel-first architecture or data control, SigNoz is a serious answer. If they're minimal operational load and maximum out-of-the-box coverage, a managed platform is. Neither choice closes the gap this article is about.

Why instrumentation doesn't close the gap

After the OTel pipelines are flowing, the Services tab is green, and the alert rules are tuned, an incident still unfolds the same way. An alert fires, a person context-switches, opens SigNoz, runs the steps above, checks whatever deployed recently, and decides what to do. The telemetry made that person faster. It didn't remove the twenty to forty minutes of attention per incident.

The reasons are structural, not a matter of tuning:

Alert rules are static. A threshold encodes what "bad" looked like the day someone wrote it. It can't tell a deploy regression from a slow upstream from a GC pause, so it pages the same way for all three.
The cause usually lives outside the telemetry. SigNoz sees the slow span. It doesn't see the deploy, the database failover or the scaling event in your cloud control plane that made it slow.
The investigation lives in heads. Which route is fragile, which dependency flaps, which dashboard to open first: that knowledge sits with whoever has been on call longest.

SigNoz's scope ends at delivery. The alert reaches Slack or PagerDuty, and everything after that is yours. The rest of this series is about what happens on the other side of that handoff.

Where this series goes next

Part two gets everything possible out of SigNoz's native automation: alert rules across all five types, notification channels, webhook payloads and rule audits. Part three covers the step native tooling can't take. CloudThinker connects to SigNoz read-only, picks up the firing alert, runs the trace investigation the way an engineer would, correlates with deploys and cloud state, and proposes remediation with evidence. It acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Try CloudThinker free with 100 premium credits and no card required.