Langfuse LLM Observability: 6 Failure Modes Your Logs Won't Catch

Someone tightened a system prompt on Tuesday. Two sentences, reviewed in ten minutes, shipped with green tests. For the next week your assistant gave worse answers to a whole category of questions. No exceptions, no 500s, no alert. You found out the following Thursday, from a customer email.

That's the defining problem of LLM observability, and it's why teams adopt Langfuse: LLM features fail silently. A classic web service fails loudly, with stack traces, error rates and pages. An LLM feature returns HTTP 200 with a worse answer, a longer wait or a bigger token bill, and nothing in your APM stack notices. Langfuse captures the raw material to catch this (traces, generations, scores, prompt versions, token costs), but only if you know which failure modes to look for.

This guide covers the six that show up on nearly every team shipping LLM features: what each looks like, why it happens, one concrete way to detect it in Langfuse today, and what it typically costs you in money or debugging hours.

Part two is a hands-on guide to Langfuse's native tooling: tracing, prompt management, evals and dashboards. Part three covers putting an AI agent on top of your Langfuse data so regressions get investigated without a human staring at dashboards.

01Quality

Silent regression

A prompt edit makes answers worse for one slice of inputs. Every request still returns 200.

02Cost

Token creep

Spend grows faster than usage, and the invoice has one line per month.

03Latency

Chain stacking

Five calls at under two seconds each add up to a nine-second wait.

04Review

Unread traces

Everything is recorded and nobody opens it until a customer complains.

05Prompts

Scattered versions

f-strings, config files and notebooks, with no record of what ran on a given day.

06Evals

Gate nothing

The eval set exists, runs after releases, and blocks nothing. This one feeds the first.

None of the six throws an error. All six are drift: quality, cost, latency or process moving away from where you left it.

Why your existing monitoring can't see these

Your APM stack was built around a simple contract: a request either succeeds or fails, and failure has a shape (an exception, a status code, a timeout). Error rate and latency percentiles cover most of what can go wrong with a database call or an HTTP handler.

LLM features break that contract in three structural ways.

Correctness isn't in the response code. The model returns well-formed text whether the answer is right or wrong. The only way to know an answer got worse is to score it, and APM has no concept of a score.
The behavior lives in data, not code. A prompt, a few-shot example, a retrieval chunk size or a model version can change outputs without a deploy touching the service. Your change log and your behavior log drift apart.
Cost is per call and per token, not per host. A retry loop that doubles your bill looks like a healthy, busy service on every infrastructure dashboard.

So the signals that matter (score per prompt version, cost per feature, latency per span in a chain) have to come from LLM-aware tracing. Langfuse records them. The six failure modes below are what you find when you start reading them.

	Classic service failure	LLM feature failure
What the user sees	An error page	A plausible, worse answer
What the monitor sees	5xx rate, exception count	Nothing, unless you score outputs
What caused it	A code deploy	A prompt, model, retrieval or input change
When you find out	Minutes, from a page	Days, from a customer
Where the evidence is	Logs and traces	Traces, scores and prompt versions together

1. Silent quality regressions after a prompt change

What happens. A prompt edit changes model behavior in ways nobody predicted: the "be concise" instruction that makes the model drop citations, or the reordered few-shot examples that break an edge case. There's no stack trace because nothing crashed. Output quality just drops, for some slice of inputs, until a human notices.

Why it happens. Prompts are code that isn't treated like code. Most teams have no diff review that catches semantic drift, no canary, and no metric that moves when answers get worse. The feedback loop is customer complaints, which arrive days later and arrive angry.

How to detect it. You need a quality signal per trace (user feedback thumbs, an LLM-as-a-judge score, or a heuristic check) recorded as a Langfuse score. Once scores exist, regression detection is a before and after comparison around the change. In the UI, open Dashboards, chart your score average over time, and mark the day the prompt shipped. If you manage the prompt in Langfuse and link it to generations, each generation records the prompt version that produced it, so you can group scores by version instead of by date.

Typical cost. The regression itself varies, but the debugging pattern is consistent: without per-version scores, teams typically burn one to three engineer-days reconstructing "what changed and when" from git history and guesswork. With scores tied to prompt versions, the same question takes minutes.

2. Token cost creep per feature and per user

What happens. Your LLM bill grows faster than usage. A retry loop that silently doubled calls, a RAG pipeline stuffing 30 chunks into context where 8 would do, one integration customer hammering your most expensive endpoint, a model upgrade that raised the per-token price. Each one is invisible in an aggregate invoice.

Why it happens. Providers bill by the token, and your finance team sees one number per month. Without attribution to feature, user and model, nobody can say which of your four LLM features caused June's 40% jump.

How to detect it. Langfuse computes usage and cost per generation automatically for known models. Pull daily cost split by trace name. Name your traces after features; it's the single highest-leverage instrumentation decision you'll make.

curl -s "https://cloud.langfuse.com/api/public/metrics/daily?traceName=support-chat" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | jq '.data[] | {date, totalCost}'

Repeat per feature, or add userId=... to the query string to isolate a single tenant. Run it once a week and cost creep stops being a quarterly surprise. If you self-host, swap the host for your own Langfuse URL.

Typical cost. Teams doing their first per-feature cost breakdown commonly find 20 to 40% of spend attributable to something unintentional: oversized context, redundant calls, or one runaway consumer.

3. Latency stacking across chained calls

What happens. No single LLM call is slow, but the user waits nine seconds. Modern features are chains: rewrite the query, embed it, retrieve, rerank, generate, maybe a guardrail check on the way out. Five sequential calls at 1.5 to 2 seconds each add up to an experience people abandon.

Why it happens. Each call was benchmarked alone, at build time, with a warm cache. Nobody measures the end-to-end trace in production, where the retrieval step got slower as the index grew and someone added "one more" validation call in March.

How to detect it. This is exactly what a trace waterfall is for. In Langfuse, open Tracing → Traces, sort by latency descending, open one of the top offenders, and read the nested spans. The waterfall shows which step dominates and whether calls that could run in parallel are running one after another. For the aggregate view, chart latency percentiles per observation name in Dashboards. Watch p95, not the average.

Typical cost. The usual finding is that one step accounts for half the wall-clock time, and that at least one pair of sequential calls has no data dependency and could run in parallel. That's often a 30 to 50% end-to-end latency cut for an afternoon of work.

4. Traces nobody reviews until a customer complains

What happens. You did the hard part. Everything is instrumented and traces flow into Langfuse. Then nobody opens them. The data sits there until a complaint arrives, at which point someone searches for that user's session and finds the failure had been happening for two weeks.

Why it happens. Dashboards are pull, not push. Reviewing traces has no owner, no cadence and no trigger. On a team shipping features, "look at traces" loses to everything with a deadline, every time.

How to detect it (or rather, make review happen at all). Pick one filtered view and put a 15-minute weekly slot on it. The highest-yield filter is negative user feedback: in Tracing → Traces, filter by your feedback score (for example user-feedback below 1) over the last 7 days and read five traces end to end. Five bad traces a week teach you more about your failure modes than any aggregate chart.

Typical cost. Measured in detection latency: the gap between "first bad trace" and "a human sees it" is typically one to three weeks on teams with no review cadence. Every debugging session also starts colder, reproducing from a complaint instead of reading the trace that already recorded everything.

5. Prompt versions scattered across the codebase

What happens. The production prompt lives in a Python f-string. A variant lives in a config file. The one the intern improved lives in a notebook. When quality drops, "what prompt was actually running on June 3rd?" means git archaeology across three repos, and if prompts are assembled at runtime, git may not even have the answer.

Why it happens. Prompts start as string literals because that's the fastest way to ship the prototype. Nothing forces the migration to managed versioning, so entropy wins.

How to detect it. Two checks. First, list what Langfuse knows about:

curl -s "https://cloud.langfuse.com/api/public/v2/prompts" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | jq '.data[] | {name, labels}'

Then grep your codebase for prompt-shaped string literals that bypass it:

grep -rn --include='*.py' --include='*.ts' -E '"(You are|Act as|Your task)' src/ | grep -v get_prompt

Anything the second command finds and the first doesn't list is an unmanaged prompt. Langfuse prompt management gives you versions, labels (production, staging) and instant rollback, and, combined with failure mode 1, per-version quality scores.

Typical cost. Every incident involving an unmanaged prompt starts with an hour of forensics before debugging can begin. Rollback, the correct first response to a prompt regression, becomes a code deploy instead of a label change.

6. Evals that exist but don't gate anything

What happens. The team built an eval set: fifty curated cases, an LLM-as-a-judge rubric, maybe a Langfuse dataset. It ran twice, both times by hand, both times after a release. Prompt changes still ship on the author's judgment, and the eval is decoration.

Why it happens. Wiring evals into the release path is unglamorous integration work, and there's always a reason to skip it "just this once." An eval that blocks nothing decays fast, because nothing breaks when it's stale.

How to detect it. Ask yourself one question. Open Datasets in Langfuse and look at the run history for your main dataset. If the last dataset run predates your last prompt change, your evals gate nothing. The fix is mechanical: run the experiment against every prompt candidate before it gets the production label, and record scores to the run. Part two of this series walks through it.

Typical cost. This is the failure mode that causes failure mode 1. Teams with ungated evals ship quality regressions at whatever rate their prompt-change velocity implies. Teams that gate catch most of them before release, for a few minutes of eval runtime.

The pattern behind all six

None of these produce an error. All six are drift: quality, cost, latency or process moving away from where you left it, at whatever pace your team ships. Langfuse's job is to record the evidence: every trace, every generation with its token counts and cost, every score, every prompt version. Your job is to actually look.

01Daily

Cost per feature

Metrics API split by trace name. Catches retry loops and runaway tenants within a day.

02Weekly

p95 and bad traces

Latency percentiles per observation, plus five negative-feedback traces read end to end.

03Every prompt change

Eval run

A dataset run against the candidate before it gets the production label.

The minimum cadence that covers all six failure modes. The third row is the one that prevents regressions instead of finding them.

That cadence is cheap on paper and hard in practice, because it competes with shipping. The gap between "the data is in Langfuse" and "someone acted on it" is where all six failure modes live.

Where this series goes next

Part two closes that gap the manual way, with Langfuse's own tooling: sessions and nested spans, prompt labels, scores and LLM-as-a-judge, dashboards, and the honest limits of native alerting. Part three asks the obvious follow-up: why is a human doing the watching at all?

CloudThinker connects to Langfuse read-only and runs that watch continuously. It tracks cost per feature, latency percentiles and score trends, and when something regresses it investigates (diffs the prompt versions, samples the bad traces, lines the window up with what shipped) and brings you a finding instead of a chart. Changes go through Auto Mode, so you decide what runs on its own and what waits for approval.