LLM Observability + Evals
Use this skill to make LLM and agent behavior measurable. It routes across installed observability skills and provides the evaluation loop that connects traces, datasets, scorers, and regressions.
Routing
- Use
langfuse for Langfuse tracing, prompt management, datasets, and production observability.
- Use
phoenix-cli and phoenix-evals for Phoenix tracing/evaluation workflows.
- Use this skill when deciding what to instrument, how to score behavior, or how to turn traces into tests.
Workflow
- Define the behavior to measure:
- task success
- factuality
- tool correctness
- latency
- cost
- refusal/guardrail behavior
- Instrument traces around meaningful spans:
- user request
- planner/router decisions
- retrieval
- tool calls
- model calls
- final answer
- Capture inputs, outputs, model/provider, token counts, latency, tool arguments, and error states.
- Build eval datasets from:
- known regression cases
- sampled production traces
- synthetic adversarial cases
- golden examples from domain experts
- Add scorers:
- deterministic checks for schemas, citations, or tool outcomes
- LLM-as-judge only with rubrics and calibration examples
- human review for high-risk workflows
- Run evals before prompt, model, tool, or retrieval changes.
Reporting
Report pass rate, cost, latency, sample size, scorer definitions, confidence limits where relevant, and the exact prompt/model/tool versions under test.
1---2name: llm-observability-evals3description: LLM and agent observability, tracing, and evaluation workflows with langfuse, phoenix-cli, and phoenix-evals. Use when instrumenting Langfuse, Phoenix, OpenTelemetry GenAI traces, eval datasets, prompt experiments, latency/cost debugging, trace scoring, or regression testing agent behavior.4metadata:5 source: https://skillmd.com/skills/stanfish06/llm-observability-evals6---78# LLM Observability + Evals910Use this skill to make LLM and agent behavior measurable. It routes across installed observability skills and provides the evaluation loop that connects traces, datasets, scorers, and regressions.1112## Routing1314- Use `langfuse` for Langfuse tracing, prompt management, datasets, and production observability.15- Use `phoenix-cli` and `phoenix-evals` for Phoenix tracing/evaluation workflows.16- Use this skill when deciding what to instrument, how to score behavior, or how to turn traces into tests.1718## Workflow19201. Define the behavior to measure:21 - task success22 - factuality23 - tool correctness24 - latency25 - cost26 - refusal/guardrail behavior272. Instrument traces around meaningful spans:28 - user request29 - planner/router decisions30 - retrieval31 - tool calls32 - model calls33 - final answer343. Capture inputs, outputs, model/provider, token counts, latency, tool arguments, and error states.354. Build eval datasets from:36 - known regression cases37 - sampled production traces38 - synthetic adversarial cases39 - golden examples from domain experts405. Add scorers:41 - deterministic checks for schemas, citations, or tool outcomes42 - LLM-as-judge only with rubrics and calibration examples43 - human review for high-risk workflows446. Run evals before prompt, model, tool, or retrieval changes.4546## Reporting4748Report pass rate, cost, latency, sample size, scorer definitions, confidence limits where relevant, and the exact prompt/model/tool versions under test.