Sign inBook a Demo

How To

Langfuse LLM Observability: 6 Failure Modes Your Logs Won't Catch

Langfuse LLM observability, explained through the six failure modes that hit every team shipping LLM features: silent quality regressions after a prompt change, token cost creep per feature and user, latency stacking across chained calls, traces nobody reviews, prompt versions scattered across the codebase, and evals that don't gate releases. For each: why it happens, one concrete detection step using Langfuse traces, scores, cost tracking, or prompt management, and what it typically costs in money or debugging hours. Part one of our LLM observability series for AI engineering teams.

ST
·
langfuseobservabilityllmopsllmtracingpromptmanagementaiengineering
Cover Image for Langfuse LLM Observability: 6 Failure Modes Your Logs Won't Catch

Langfuse LLM Observability: 6 Failure Modes Your Logs Won't Catch

Someone tightened a system prompt on Tuesday — two sentences, reviewed in ten minutes, shipped with green tests. For the next week your assistant gave measurably worse answers to a whole category of questions. No exceptions, no 500s, no alert. You found out on the following Thursday, from a customer email.

That's the defining problem of LLM observability, and it's why teams adopt Langfuse: LLM features fail silently. A classic web service fails loudly — stack traces, error rates, pager noise. An LLM feature returns HTTP 200 with a worse answer, a longer wait, or a bigger token bill, and nothing in your APM stack notices. Langfuse captures the raw material to catch this — traces, generations, scores, prompt versions, token costs — but only if you know which failure modes to look for.

This guide covers the six that show up on nearly every team shipping LLM features: what each one looks like, why it happens, one concrete way to detect it in Langfuse today, and what it typically costs you in money or debugging hours.

This is part one of a three-part series. Part two is a hands-on guide to Langfuse's native tooling — tracing, prompt management, evals, dashboards; part three covers putting an AI agent on top of your Langfuse data so regressions get investigated without a human staring at dashboards.

1. Silent quality regressions after a prompt change

What happens. A prompt edit changes model behavior in ways nobody predicted — the "be concise" instruction that makes the model drop citations, the reordered few-shot examples that break an edge case. There is no stack trace because nothing crashed. Output quality just drops, for some slice of inputs, until a human notices.

Why it happens. Prompts are code that isn't treated like code. Most teams have no diff review that catches semantic drift, no canary, and no metric that moves when answers get worse. The feedback loop is customer complaints, which arrive days later and arrive angry.

How to detect it. You need a quality signal per trace — user feedback thumbs, an LLM-as-judge score, or a heuristic check — recorded as a Langfuse score. Once scores exist, regression detection is a before/after comparison around the change. In the UI: Dashboards, chart your score average over time, and annotate the day the prompt shipped. If you manage the prompt in Langfuse, the generation metadata tells you exactly which prompt version produced each trace, so you can group scores by version instead of by date.

Typical cost. The regression itself varies, but the debugging pattern is consistent: without per-version scores, teams typically burn one to three engineer-days reconstructing "what changed and when" from git history and vibes. With scores tied to prompt versions, the same question takes minutes.

2. Token cost creep per feature and per user

What happens. Your LLM bill grows faster than usage. A retry loop that silently doubled calls, a RAG pipeline stuffing 30 chunks into context where 8 would do, one integration customer hammering your most expensive endpoint, a model upgrade that raised per-token price — each invisible in an aggregate invoice.

Why it happens. Providers bill by the token; your finance team sees one number per month. Without attribution to feature, user, and model, the invoice is unfalsifiable. Nobody can say which of your four LLM features caused June's 40% jump.

How to detect it. Langfuse computes usage and cost per generation automatically for known models. Pull daily cost split by trace name (name your traces after features — this is the single highest-leverage instrumentation decision you'll make):

curl -s "https://cloud.langfuse.com/api/public/metrics/daily?traceName=support-chat" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | jq '.data[] | {date, totalCost}'

Repeat per feature, or add userId=... to the query string to isolate a single tenant. Do this once a week and cost creep stops being a quarterly surprise.

Typical cost. Teams doing their first per-feature cost breakdown commonly find 20–40% of spend attributable to something unintentional — oversized context, redundant calls, or one runaway consumer.

3. Latency stacking across chained calls

What happens. No single LLM call is slow, but the user waits nine seconds. Modern features are chains: rewrite the query, embed it, retrieve, rerank, generate, maybe a guardrail check on the way out. Five sequential calls at 1.5–2 seconds each add up to an experience people abandon.

Why it happens. Each call was benchmarked alone, at build time, with a warm cache. Nobody measures the end-to-end trace in production, where the retrieval step got slower as the index grew and someone added "one more" validation call in March.

How to detect it. This is exactly what a trace waterfall is for. In Langfuse: Tracing → Traces, sort by latency descending, open one of the top offenders, and read the nested spans — the waterfall shows which step dominates and whether calls that could run in parallel are running sequentially. For the aggregate view, chart latency percentiles per observation name in Dashboards; p95 is the number to watch, not the average.

Typical cost. The usual finding is that one step accounts for half the wall-clock time, and that at least one pair of sequential calls has no data dependency and could be parallelized — often a 30–50% end-to-end latency cut for an afternoon of work.

4. Traces nobody reviews until a customer complains

What happens. You did the hard part — instrumented everything, traces flow into Langfuse — and then nobody opens them. The data sits there until a complaint arrives, at which point someone searches for that user's session and finds the failure was happening for two weeks.

Why it happens. Dashboards are pull, not push. Reviewing traces has no owner, no cadence, and no trigger. On a team shipping features, "look at traces" loses to everything with a deadline, every time.

How to detect it — or rather, to make review happen at all: pick one filtered view and put a 15-minute weekly slot on it. The highest-yield filter is negative user feedback: in Tracing → Traces, filter by your feedback score (e.g., user-feedback below 1) over the last 7 days and read five traces end to end. Five bad traces a week teaches you more about your failure modes than any aggregate chart.

Typical cost. Measured in detection latency: the gap between "first bad trace" and "human sees it" is typically one to three weeks on teams with no review cadence. Every debugging session also starts colder — reproducing from a complaint instead of reading the trace that already recorded everything.

5. Prompt versions scattered across the codebase

What happens. The production prompt lives in a Python f-string. A variant lives in a config file. The one the intern improved lives in a notebook. When quality drops, "what prompt was actually running on June 3rd?" requires git archaeology across three repos — and if prompts are interpolated at runtime, git may not even have the answer.

Why it happens. Prompts start as string literals because that's the fastest way to ship the prototype. Nothing forces the migration to managed versioning, so entropy wins.

How to detect it. Two checks. First, inventory what Langfuse knows about:

curl -s "https://cloud.langfuse.com/api/public/v2/prompts" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | jq '.data[].name'

Then grep your codebase for prompt-shaped string literals that bypass it:

grep -rn --include='*.py' --include='*.ts' -E '"(You are|Act as|Your task)' src/ | grep -v get_prompt

Anything the second command finds and the first doesn't list is an unmanaged prompt. Langfuse prompt management gives you versions, labels (production, staging), and instant rollback — and, combined with failure mode #1, per-version quality scores.

Typical cost. Every incident involving an unmanaged prompt starts with an hour of forensics before debugging can begin. Rollback — the correct first response to a prompt regression — becomes a code deploy instead of a label change.

6. Evals that exist but don't gate anything

What happens. The team built an eval set — fifty curated cases, an LLM-as-judge rubric, maybe a Langfuse dataset. It ran twice, both times manually, both times after a release. Prompt changes still ship on the author's judgment; the eval is decoration.

Why it happens. Wiring evals into the release path is unglamorous integration work, and there's always a reason to skip it "just this once." An eval that doesn't block anything decays fast, because nothing breaks when it's stale.

How to detect it. Audit yourself with one question: open Datasets in Langfuse and look at the run history for your main dataset. If the last dataset run predates your last prompt change, your evals gate nothing. The fix is mechanical — run the experiment against every prompt candidate before it gets the production label, and record scores to the run — and part two of this series walks through it.

Typical cost. This is the failure mode that causes failure mode #1. Teams with ungated evals ship quality regressions at whatever rate their prompt-change velocity implies; teams that gate catch most of them pre-release for the cost of a few minutes of eval runtime.

The pattern behind all six

Look at the list again: none of these produce an error. All six are drift — quality, cost, latency, or process drifting away from where you left it, at whatever pace your team ships. Langfuse's job is to record the evidence: every trace, every generation with its token counts and cost, every score, every prompt version. Your job is to actually look — daily at cost per feature, weekly at p95 latency and negative-feedback traces, on every prompt change at eval scores.

The gap between "the data is in Langfuse" and "someone acted on it" is where all six failure modes live. Part two closes that gap the manual way, with Langfuse's own tooling. Part three asks the obvious follow-up: why is a human doing this at all?

Watching the watchtower, continuously

Everything above assumes an engineer checks the dashboards. CloudThinker connects to Langfuse read-only and puts agents on that job: they watch cost per feature, latency percentiles, and score trends across your traces, and when something regresses they investigate — diff the prompt versions, sample the bad traces, correlate with the deploy — and bring you findings instead of charts. Nothing changes without your approval. Try CloudThinker free — 100 premium credits, no card required.