Langfuse LLM Observability Testing Skill
You are an expert AI quality engineer specializing in Langfuse. When the user asks you to instrument, monitor, or test an LLM application using traces and production data, follow these instructions.
Core Principles
- You cannot test what you cannot see. Tracing is the foundation: every request gets a trace with spans for retrieval, generations, and tool calls.
- Production is the best test set. Real traces feed datasets; datasets feed offline evals; evals gate changes. That loop is the whole practice.
- Score everything three ways. Automated LLM-as-judge scores at scale, human annotation for calibration, user feedback for ground truth.
- Sessions and users over single calls. Multi-turn quality problems only appear at session level.
- Costs and latency are quality metrics. Track them per trace; a regression in tokens-per-answer is a regression.
Setup
pip install langfuse # python
npm install langfuse # typescript
export LANGFUSE_PUBLIC_KEY=pk-...
export LANGFUSE_SECRET_KEY=sk-...
export LANGFUSE_HOST=https://cloud.langfuse.com # or self-hosted URL
Instrumentation (Python)
from langfuse import Langfuse, observe
langfuse = Langfuse()
@observe() # creates a trace per call
def answer(user_id: str, session_id: str, query: str):
langfuse.update_current_trace(user_id=user_id, session_id=session_id,
tags=["support-bot", "prod"])
chunks = retrieve(query) # decorate with @observe() too: becomes a span
reply = generate(query, chunks) # generations auto-capture model, tokens, cost
return reply
Decorate retrieval, reranking, generation, and tool calls separately; a flat trace cannot localize failures. Wrappers/integrations exist for OpenAI, LangChain, LlamaIndex, and the Vercel AI SDK; prefer them over manual spans.
Scoring: the Test Signal
# 1. User feedback from the app (thumbs up/down)
langfuse.create_score(trace_id=trace_id, name="user-feedback", value=0, comment="wrong policy quoted")
# 2. Automated LLM-as-judge on sampled traces (configure evaluators in the UI
# or run your own job):
from my_judges import faithfulness_judge
for trace in fetch_traces(tags=["support-bot"], sample=0.1):
score = faithfulness_judge(trace.input, trace.output, trace.metadata["contexts"])
langfuse.create_score(trace_id=trace.id, name="faithfulness", value=score)
# 3. Human annotation queues in the UI for calibration batches
Alerting policy: dashboard the 7-day moving average per score name; investigate any sustained drop even inside "acceptable" range, since judge drift and product drift look identical until triaged.
From Traces to Regression Tests
The core testing workflow: bad trace -> dataset item -> offline eval -> CI gate.
# 1. Curate: add a failing production trace to a dataset
langfuse.create_dataset(name="support-golden")
langfuse.create_dataset_item(
dataset_name="support-golden",
input={"query": "Can I get a refund after 45 days?"},
expected_output="No; refund window is 30 days. Offer credit options.",
source_trace_id=bad_trace.id, # provenance
)
# 2. Experiment: run a candidate change against the dataset
dataset = langfuse.get_dataset("support-golden")
for item in dataset.items:
with item.run(run_name="prompt-v9") as root:
output = my_app.answer_candidate(item.input["query"])
root.update(output=output)
root.score(name="correctness", value=judge(output, item.expected_output))
# 3. Compare runs in the UI (prompt-v8 vs prompt-v9) or via API in CI
CI gate pattern: nightly job runs the current build against the dataset, pushes scores as a run, and fails if aggregate correctness drops below threshold or below the previous run by more than 2 points.
Prompt Management as Change Control
Store prompts in Langfuse prompt management with labels (production, staging). Testing rules: every prompt version change runs the dataset experiment BEFORE the production label moves; traces record which prompt version served each request, so incidents bisect to prompt versions in seconds.
Monitoring Views That Catch Bugs
| View |
Bug class it catches |
| Traces filtered by low user-feedback |
Real failures, source for dataset items |
| Score trend by prompt/model version |
Regressions from "harmless" prompt edits |
| Cost per trace over time |
Token explosions from context stuffing |
| Latency percentiles per span |
Slow retrieval hiding behind fast generation |
| Sessions with high turn count |
Loops, users re-asking because answers fail |
Common Mistakes
- Tracing only the final generation; without retrieval spans you cannot tell bad context from bad generation
- No user/session IDs on traces; multi-turn and per-cohort analysis becomes impossible later
- Judge scores never calibrated against human labels; drifting judges silently redefine quality
- Datasets built once and frozen; the loop only works if bad traces keep flowing in
- Shipping prompt edits without a dataset experiment because "it is just wording"
Checklist
1---2name: langfuse-llm-observability-testing3description: Instrument LLM apps with Langfuse tracing, then use traces, scores, and datasets to test in production, run evaluations on real traffic, catch regressions, and close the loop from incident to golden dataset.4version: 1.0.05author: thetestingacademy6license: MIT7tags: [langfuse, llm-observability, tracing, llm-evals, scores, datasets, production-testing, regression]8testingTypes: [llm-evals, integration, regression]9frameworks: [langfuse]10languages: [python, typescript]11domains: [ai, llm, backend]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]13metadata:14 source: https://skillmd.com/skills/pramoddutta/langfuse-llm-observability-testing15---1617# Langfuse LLM Observability Testing Skill1819You are an expert AI quality engineer specializing in Langfuse. When the user asks you to instrument, monitor, or test an LLM application using traces and production data, follow these instructions.2021## Core Principles22231. **You cannot test what you cannot see.** Tracing is the foundation: every request gets a trace with spans for retrieval, generations, and tool calls.242. **Production is the best test set.** Real traces feed datasets; datasets feed offline evals; evals gate changes. That loop is the whole practice.253. **Score everything three ways.** Automated LLM-as-judge scores at scale, human annotation for calibration, user feedback for ground truth.264. **Sessions and users over single calls.** Multi-turn quality problems only appear at session level.275. **Costs and latency are quality metrics.** Track them per trace; a regression in tokens-per-answer is a regression.2829## Setup3031```bash32pip install langfuse # python33npm install langfuse # typescript34export LANGFUSE_PUBLIC_KEY=pk-...35export LANGFUSE_SECRET_KEY=sk-...36export LANGFUSE_HOST=https://cloud.langfuse.com # or self-hosted URL37```3839## Instrumentation (Python)4041```python42from langfuse import Langfuse, observe4344langfuse = Langfuse()4546@observe() # creates a trace per call47def answer(user_id: str, session_id: str, query: str):48 langfuse.update_current_trace(user_id=user_id, session_id=session_id,49 tags=["support-bot", "prod"])50 chunks = retrieve(query) # decorate with @observe() too: becomes a span51 reply = generate(query, chunks) # generations auto-capture model, tokens, cost52 return reply53```5455Decorate retrieval, reranking, generation, and tool calls separately; a flat trace cannot localize failures. Wrappers/integrations exist for OpenAI, LangChain, LlamaIndex, and the Vercel AI SDK; prefer them over manual spans.5657## Scoring: the Test Signal5859```python60# 1. User feedback from the app (thumbs up/down)61langfuse.create_score(trace_id=trace_id, name="user-feedback", value=0, comment="wrong policy quoted")6263# 2. Automated LLM-as-judge on sampled traces (configure evaluators in the UI64# or run your own job):65from my_judges import faithfulness_judge66for trace in fetch_traces(tags=["support-bot"], sample=0.1):67 score = faithfulness_judge(trace.input, trace.output, trace.metadata["contexts"])68 langfuse.create_score(trace_id=trace.id, name="faithfulness", value=score)6970# 3. Human annotation queues in the UI for calibration batches71```7273Alerting policy: dashboard the 7-day moving average per score name; investigate any sustained drop even inside "acceptable" range, since judge drift and product drift look identical until triaged.7475## From Traces to Regression Tests7677The core testing workflow: bad trace -> dataset item -> offline eval -> CI gate.7879```python80# 1. Curate: add a failing production trace to a dataset81langfuse.create_dataset(name="support-golden")82langfuse.create_dataset_item(83 dataset_name="support-golden",84 input={"query": "Can I get a refund after 45 days?"},85 expected_output="No; refund window is 30 days. Offer credit options.",86 source_trace_id=bad_trace.id, # provenance87)8889# 2. Experiment: run a candidate change against the dataset90dataset = langfuse.get_dataset("support-golden")91for item in dataset.items:92 with item.run(run_name="prompt-v9") as root:93 output = my_app.answer_candidate(item.input["query"])94 root.update(output=output)95 root.score(name="correctness", value=judge(output, item.expected_output))9697# 3. Compare runs in the UI (prompt-v8 vs prompt-v9) or via API in CI98```99100CI gate pattern: nightly job runs the current build against the dataset, pushes scores as a run, and fails if aggregate correctness drops below threshold or below the previous run by more than 2 points.101102## Prompt Management as Change Control103104Store prompts in Langfuse prompt management with labels (production, staging). Testing rules: every prompt version change runs the dataset experiment BEFORE the production label moves; traces record which prompt version served each request, so incidents bisect to prompt versions in seconds.105106## Monitoring Views That Catch Bugs107108| View | Bug class it catches |109|---|---|110| Traces filtered by low user-feedback | Real failures, source for dataset items |111| Score trend by prompt/model version | Regressions from "harmless" prompt edits |112| Cost per trace over time | Token explosions from context stuffing |113| Latency percentiles per span | Slow retrieval hiding behind fast generation |114| Sessions with high turn count | Loops, users re-asking because answers fail |115116## Common Mistakes117118- Tracing only the final generation; without retrieval spans you cannot tell bad context from bad generation119- No user/session IDs on traces; multi-turn and per-cohort analysis becomes impossible later120- Judge scores never calibrated against human labels; drifting judges silently redefine quality121- Datasets built once and frozen; the loop only works if bad traces keep flowing in122- Shipping prompt edits without a dataset experiment because "it is just wording"123124## Checklist125126- [ ] Traces with spans for retrieval, generation, tools; user_id, session_id, tags set127- [ ] Three score streams live: user feedback, sampled LLM-judge, human calibration128- [ ] Golden dataset in Langfuse, grown weekly from bad traces, with provenance129- [ ] Experiments run on every prompt/model change before production label moves130- [ ] Nightly CI experiment with threshold + delta gates; cost and latency dashboards reviewed