Langfuse Evaluation

Langfuse Evaluation

This skill carries the durable judgment of evaluating LLM applications with Langfuse: how the evaluation loop works, which method to use when, and how scores, datasets, and experiments fit together. It does not embed SDK code — Langfuse updates frequently, so fetch current code from live docs and hand calibration/CI specifics to the vendored langfuse skill.

Operating principles

1. Distill judgment, fetch facts. This skill owns the decisions and workflow. For exact code (creating scores, defining tasks/evaluators, running experiments), fetch the live doc by appending .md to the page URL (e.g. https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk.md) or use the SDK references (python.reference.langfuse.com, js.reference.langfuse.com). Never write eval code from memory.
2. Defer to the vendored langfuse skill for: judge calibration / reliability (skills/langfuse/references/judge-calibration.md), CI/CD experiment gates (skills/langfuse/references/ci-cd.md), systematic error analysis (skills/langfuse/references/error-analysis.md), and capturing user feedback as scores (skills/langfuse/references/user-feedback.md). Don't duplicate those here.

Workflow

1. Frame the evaluation

First decide offline or online, and what the score attaches to:

• Offline = gate a change before deploy → datasets + experiments (references/datasets-experiments.md).
• Online = score live production traces → automated evaluators on traces.
• Most "how do I evaluate X" questions resolve to those two axes. See references/methods-overview.md.

2. Choose the method

Start from references/methods-overview.md to pick among the five methods by what's being judged (deterministic vs subjective) and scale; methods compose. Then go deep in the method reference:

• references/llm-as-a-judge.md — subjective judgment at scale (setup, where to point it; calibration deferred to the vendored skill).
• references/code-evaluators.md — deterministic checks (JSON/schema/match/business rules).
• references/human-annotation.md — annotation queues + human-in-the-loop scoring from your own tool (ground truth).

3. Model the scores

Every method emits a score. Get the level (trace/observation/session/dataset-run) and data type (NUMERIC/CATEGORICAL/BOOLEAN/TEXT) right, and use a ScoreConfig for any score multiple people/pipelines produce. See references/scores.md — note TEXT scores can't be aggregated.

4. Build datasets & run experiments (offline)

Follow references/datasets-experiments.md: assemble dataset items (seed from production traces), define task + evaluator functions, run via UI (quick) or SDK (full control), then interpret results top-down (aggregate metrics → item-level diff vs baseline → trace debugging → annotate regressions) — that section is the high-judgment payoff. Manage datasets in Langfuse for comparison views. Fetch live docs for the actual SDK code.

5. Set up online evaluation

Configure evaluators (LLM-judge / code / annotation) to score production traces automatically; feed surprising cases back into the dataset so offline experiments catch them next time.

6. Specialized application types

For these, start from the matching reference (each builds on steps 1–5, not a replacement):

• RAG → references/rag-evals.md (evaluate retrieval and generation as separate components).
• Agents → references/agent-evals.md (trajectory + per-step + final-answer; 3 phases).
• Multi-turn / conversational → references/multi-turn-evals.md (session-level; real N+1 vs simulated).
• Custom / scheduled / external-framework evaluation → references/external-pipelines.md.

Bundled resources (more references being added per the roadmap)

• references/methods-overview.md — the evaluation loop and which method when (offline/online, the five methods, how to choose, how they compose).
• references/scores.md — the universal score object: attachment levels, the four data types and when each, source, ScoreConfig schema enforcement, scores-vs-tags.
• references/datasets-experiments.md — datasets, dataset items, tasks, evaluators, experiment runs; data relationships; UI vs SDK; the local-dataset caveat; interpreting results (the top-down funnel: aggregate → item diff → trace debug → annotate).
• references/llm-as-a-judge.md — designing/setting up a judge and where to point it (observations/traces/experiments); calibration deferred to the vendored skill.
• references/code-evaluators.md — deterministic Python/TS checks; when vs a judge; where they run.
• references/human-annotation.md — annotation queues + human-in-the-loop custom-tool scoring; building ground truth.
• references/rag-evals.md — RAG: evaluate retrieval vs generation independently; chunking; faithfulness/relevancy/context metrics; Ragas (reference-free).
• references/agent-evals.md — agents: trajectory + per-step + final-answer; 3 failure modes; 3 phases; black/glass/white-box strategies.
• references/multi-turn-evals.md — conversational apps: session-level memory/coherence/ resolution; real N+1 evaluation vs simulated conversations.
• references/external-pipelines.md — when/how to evaluate outside Langfuse (scheduled/ webhook), fetch→score→ingest architecture.

Hand-off map