Langfuse LLM Observability with Native Tools: Traces, Prompts, Evals

There's a difference between logging LLM calls and instrumenting them. A log line tells you a call happened. An instrumented trace tells you which user, which session, which prompt version, which retrieval step fed it, what it cost and what came out the other end. Most teams shipping LLM features are one session_id away from a debuggable system, and don't find out until the first "the bot got worse" ticket arrives with nothing to attach to it.

This is the hands-on part of our Langfuse LLM observability series. Part one covered the failure modes: silent quality regressions after prompt changes, token cost creep, latency stacking across chained calls, prompt versions scattered through the codebase, and evals that gate nothing. This article is the build: LLM tracing, prompt management, scores, evals, dashboards and alerting, using only Langfuse's native features. It closes with one regression worked by hand, and an honest look at where the DIY ceiling sits.

01Trace

Sessions and spans

session_id, user_id and nested spans turn calls into a chain you can read.

02Version

Prompt labels

Prompts fetched at runtime by label, linked to every generation.

03Score

Feedback and judges

User thumbs, LLM-as-a-judge on a sample, datasets before release.

04Chart

Dashboards

Cost by feature and user, p95 by trace name, score by prompt version.

05Alert

The thin part

Spend alerts and prompt webhooks. Everything else is a cron job you write.

Everything below uses the Python SDK (v3, built on OpenTelemetry). Setup is three environment variables (Langfuse docs):

pip install langfuse

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted URL

---

Step 1: Instrument traces properly (sessions, users, nested spans)

The fastest path to first traces is the OpenAI drop-in wrapper. Change one import and every completion call is captured with model, tokens, latency and cost:

from langfuse.openai import openai  # instead of: import openai

That gets you isolated generations. It doesn't get you debuggable ones. Three attributes turn a pile of LLM calls into something you can investigate: sessions (group all calls in one conversation), users (attribute cost and quality per account) and nested spans (see the chain, not only the final call).

The @observe decorator builds the tree from your existing call structure:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def answer_ticket(ticket_id: str, question: str, user_id: str):
    langfuse.update_current_trace(
        session_id=f"ticket-{ticket_id}",
        user_id=user_id,
        tags=["support-bot", "v2-pipeline"],
    )
    docs = retrieve_context(question)       # becomes a nested span
    return generate_answer(question, docs)  # becomes a nested generation

@observe()
def retrieve_context(question: str):
    ...  # vector search, timed as its own span

@observe(as_type="generation")
def generate_answer(question: str, docs: list[str]):
    ...  # the LLM call itself

Now a slow response decomposes: was it 4 seconds of retrieval or 4 seconds of generation? A bad answer decomposes too: did the model fail, or did retrieval hand it the wrong documents? Without the nesting, every incident starts with re-running the request locally and guessing.

If you already run OpenTelemetry, Langfuse accepts OTLP directly and the SDK works alongside existing OTel instrumentation (tracing docs), so LLM spans land in the same trace as your HTTP and database spans.

With the classic langfuse npm package, the same shape in TypeScript looks like this:

import { Langfuse } from 'langfuse';

const langfuse = new Langfuse();
const trace = langfuse.trace({
  sessionId: `ticket-${ticketId}`,
  userId,
  tags: ['support-bot'],
});
const generation = trace.generation({
  name: 'generate-answer',
  model: 'gpt-4o-mini',
  input: messages,
});
// ... call the model ...
generation.end({ output: completion });

One naming rule pays for itself later: give each trace the name of the product feature it serves (support-chat, summarizer). Cost, latency and score views all group by trace name, and part one's per-feature cost check depends on it.

Step 2: Prompt management (get prompts out of the codebase)

The prompt string hardcoded next to your API call is why "who changed the prompt and when" is unanswerable at most companies. Langfuse's prompt management gives you versioned prompts fetched at runtime, with labels acting like deployment pointers (prompt management docs):

# Fetch whatever version currently carries the "production" label
prompt = langfuse.get_prompt("support-answer", label="production")

messages = prompt.compile(question=question, context="\n".join(docs))

The workflow that makes this pay off:

1. Create version 2 in the Langfuse UI (or with langfuse.create_prompt(...)) and label it staging.
2. Point staging traffic at it. Your staging deploy fetches label="staging", while production still resolves label="production" to version 1. Same code, different pointer.
3. Promote by moving the label. When v2 looks good, assign the production label to it in the UI. No deploy, and rollback is moving the label back.

The critical detail: link the prompt to the generation so every trace records which prompt version produced it. With the OpenAI wrapper it's one argument:

completion = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    langfuse_prompt=prompt,  # ties this generation to the exact prompt version
)

After a regression, you can now filter traces by prompt version and compare v1 against v2 in the UI, the single most useful move in any "answers got worse" investigation. The SDK caches prompts client-side, so the fetch adds no meaningful latency on the hot path.

Step 3: Scores and evals (make "worse" measurable)

Traces tell you what happened. Scores tell you whether it was any good. Langfuse scores are numeric, categorical or boolean values attached to traces or individual generations (evaluation docs). Two sources matter most in production.

User feedback. Wire your thumbs-up and thumbs-down button to a score. From the browser, use the public key only (through the TS SDK's web client). From the backend:

langfuse.create_score(
    trace_id=trace_id,
    name="user_feedback",
    value=1,            # 1 = thumbs up, 0 = thumbs down
    data_type="BOOLEAN",
    comment="resolved the ticket on first answer",
)

Even 2 to 5% of users clicking gives you a quality trendline per prompt version, which is exactly the signal a silent regression needs to stop being silent.

LLM-as-a-judge. In the Langfuse UI under Evaluations → LLM-as-a-Judge, you configure an evaluator (built-in templates cover hallucination, relevance, toxicity and correctness) that runs against a sample of incoming production traces and writes scores back automatically. Run it on 5 to 10% of traffic rather than everything. Judge calls cost real tokens, and a sample is enough to see a trend. The same evaluators run against datasets for offline testing: collect problem traces into a dataset, then score every new prompt version against it before it ever gets the production label.

The pattern to aim for: user feedback as the ground-truth trickle, LLM-as-a-judge as the high-volume proxy, datasets as the pre-release gate.

Step 4: Dashboards and cost tracking per model and feature

Langfuse computes token counts and USD cost per generation from model pricing definitions, automatically for major providers (model usage and cost docs). Custom or fine-tuned models need a pricing entry in Settings → Models.

The built-in dashboards give you cost, latency (p50, p95, p99) and score trends out of the box. Custom dashboards let you slice by what matters for accountability:

• Cost by tag. Tag traces per feature (support-bot, summarizer) and "the OpenAI bill doubled" becomes "the summarizer doubled."
• Cost by user_id. Finds the one integration customer generating 40% of your spend.
• p95 latency by trace name. Latency stacking in chains shows up here first.
• Score average by prompt version. The regression chart.

For anything programmatic, the Metrics API exposes the same aggregations (Metrics API docs):

curl -s -G "https://cloud.langfuse.com/api/public/metrics" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  --data-urlencode 'query={"view":"traces","metrics":[{"measure":"totalCost","aggregation":"sum"}],"dimensions":[{"field":"tags"}],"fromTimestamp":"2026-07-01T00:00:00Z","toTimestamp":"2026-07-13T00:00:00Z"}'

Step 5: Alerting (the thin part)

Go in knowing that alerting is the least built-out layer of the native stack. Your realistic options:

• Spend alerts on Langfuse Cloud notify you at billing thresholds. Blunt, but a real backstop for cost creep.
• Prompt webhooks fire on prompt changes (version created, label moved), so at minimum every prompt promotion lands in a Slack channel with an audit trail (webhooks docs).
• Metrics API polling. A scheduled job (cron, GitHub Actions) queries cost, p95 latency or average score deltas and posts to Slack or PagerDuty when a threshold trips. It works, but you're now writing and maintaining your own alerting service.

There's no native "page me when the hallucination score drops 15% week over week." You build that yourself or you watch dashboards.

---

Working one regression by hand

Here's what the native stack looks like under load. Your weekly review shows the support-answer judge score sliding from 0.84 to 0.69 over four days (illustrative numbers). Four readings fit that chart, and each predicts something different:

Start with the cheapest discriminator: split cost, latency and volume by prompt version on the observations view.

curl -s -G "https://cloud.langfuse.com/api/public/metrics" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  --data-urlencode 'query={"view":"observations","metrics":[{"measure":"count","aggregation":"count"},{"measure":"latency","aggregation":"p95"}],"dimensions":[{"field":"promptName"},{"field":"promptVersion"}],"fromTimestamp":"2026-07-06T00:00:00Z","toTimestamp":"2026-07-13T00:00:00Z"}'

If v14 took all traffic on the day the score broke, the prompt reading moves to the front. Open the prompt in Langfuse, compare v13 and v14 side by side, then filter traces to v14 with a judge score below 0.5 and read ten of them. Say the pattern is that answers stopped citing the knowledge-base article whenever the retrieved context ran long, and v14's diff removed a few-shot example that showed a citation. That's a hypothesis you can test: move the production label back to v13 and watch the score over the next day, or add the example back as v15 on staging and run your dataset against it.

It works, and it's the right method. It also took a person most of an afternoon, and it only started because someone was doing the weekly review that week.

The honest part: what the native stack doesn't do

Run everything above and you'll have good LLM observability, better than most teams shipping AI features today. Know where the ceiling is.

01Reading

Bad traces

A score drop is a fact, not a diagnosis. Someone still opens twenty traces.

02Correlation

Across sources

Prompt history, model changes and your deploy log live in different places.

03Diffing

Which line

The UI shows v13 next to v14. Deciding which change caused the drop is judgment.

04Watching

Nobody owns it

Dashboards get opened after a complaint, which is the failure part one described.

Langfuse shows you the bad trace; someone still has to read it. A score dropping from 0.82 to 0.64 is a fact, not a diagnosis. A person still opens twenty traces, reads the outputs and forms a hypothesis.

Correlation is manual. Was the regression the prompt change, the model version bump, the deploy that altered retrieval chunking, or a new customer with unusual documents? The evidence is spread across prompt history, traces and your deploy log, and a person joins it.

Prompt diffing is a judgment call. The UI shows v1 next to v2. Deciding which changed line caused the regression, and what the fix is, costs engineer-hours every time.

Watching is a job nobody has. Dashboards are pull-based. In practice they get checked after a complaint, which is the failure mode part one described.

The native stack is the necessary foundation. The gap is the layer that reads the traces, diffs the versions and proposes the fix without a person starting from zero.

What comes next

That gap, investigation and correlation on top of your Langfuse data, is what part three of this series covers. CloudThinker connects to Langfuse read-only, watches cost, latency and score trends continuously, and when quality regresses it diffs the prompt versions, samples the bad traces and proposes the fix. It acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Related reading

• Part one: 6 failure modes your logs won't catch
• Part three: automating Langfuse LLM observability
• LLMOps vs AgentOps
• CloudThinker Connections

Try CloudThinker free with 100 premium credits and no card required.