LLM Observability + Evals
Use this skill to make LLM and agent behavior measurable. It routes across installed observability skills and provides the evaluation loop that connects traces, datasets, scorers, and regressions.
Routing
- Use
langfuse for Langfuse tracing, prompt management, datasets, and production observability.
- Use
phoenix-cli and phoenix-evals for Phoenix tracing/evaluation workflows.
- Use this skill when deciding what to instrument, how to score behavior, or how to turn traces into tests.
Workflow
- Define the behavior to measure:
- task success
- factuality
- tool correctness
- latency
- cost
- refusal/guardrail behavior
- Instrument traces around meaningful spans:
- user request
- planner/router decisions
- retrieval
- tool calls
- model calls
- final answer
- Capture inputs, outputs, model/provider, token counts, latency, tool arguments, and error states.
- Build eval datasets from:
- known regression cases
- sampled production traces
- synthetic adversarial cases
- golden examples from domain experts
- Add scorers:
- deterministic checks for schemas, citations, or tool outcomes
- LLM-as-judge only with rubrics and calibration examples
- human review for high-risk workflows
- Run evals before prompt, model, tool, or retrieval changes.
Reporting
Report pass rate, cost, latency, sample size, scorer definitions, confidence limits where relevant, and the exact prompt/model/tool versions under test.
1---2name: llm-observability-evals3description: LLM and agent observability, tracing, and evaluation workflows with langfuse, phoenix-cli, and phoenix-evals. Use when instrumenting Langfuse, Phoenix, OpenTelemetry GenAI traces, eval datasets, prompt experiments, latency/cost debugging, trace scoring, or regression testing agent behavior.4---56# LLM Observability + Evals78Use this skill to make LLM and agent behavior measurable. It routes across installed observability skills and provides the evaluation loop that connects traces, datasets, scorers, and regressions.910## Routing1112- Use `langfuse` for Langfuse tracing, prompt management, datasets, and production observability.13- Use `phoenix-cli` and `phoenix-evals` for Phoenix tracing/evaluation workflows.14- Use this skill when deciding what to instrument, how to score behavior, or how to turn traces into tests.1516## Workflow17181. Define the behavior to measure:19 - task success20 - factuality21 - tool correctness22 - latency23 - cost24 - refusal/guardrail behavior252. Instrument traces around meaningful spans:26 - user request27 - planner/router decisions28 - retrieval29 - tool calls30 - model calls31 - final answer323. Capture inputs, outputs, model/provider, token counts, latency, tool arguments, and error states.334. Build eval datasets from:34 - known regression cases35 - sampled production traces36 - synthetic adversarial cases37 - golden examples from domain experts385. Add scorers:39 - deterministic checks for schemas, citations, or tool outcomes40 - LLM-as-judge only with rubrics and calibration examples41 - human review for high-risk workflows426. Run evals before prompt, model, tool, or retrieval changes.4344## Reporting4546Report pass rate, cost, latency, sample size, scorer definitions, confidence limits where relevant, and the exact prompt/model/tool versions under test.
stanfish06