How do I install the Eval Triage skill?

Run npx skillmds@latest add langwatch/eval-triage in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Eval Triage safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Eval Triage?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Eval Triage free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under MIT.

Who published Eval Triage?

langwatch (@langwatch) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Eval Triage by langwatch — Agent Skill

Name: Eval Triage
Rating: 4.2 (5 reviews)
Author: langwatch

Eval Triage

Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.

langwatch 1f54eaf 3.4 KB Updated Sep 10, 2026

File contents

Triage Failing Experiments and Evaluations

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

Prerequisites

Step 1: Find the Failing Run

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

A run can fail two ways, and they triage differently:

Execution failure: the run errored out or stalled. status shows the error; jump to Step 4.
Score regression: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

Step 2: Isolate the Failing Rows

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

--filter failed keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
--evaluator <name> shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

Step 3: Inspect the Evaluators

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

Before blaming the agent, rule out the scorer:

LLM-judge evaluators: check the model in settings. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
Thresholds: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
Deterministic evaluators (exact match, JSON validity): these don't drift; failures here are real.

Step 4: Root Cause

Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)? git log on prompts and agent code usually answers this directly.
If rows fail on retrieval or context: inspect a production trace of the same path (langwatch trace search / langwatch trace get; see the debug-with-langwatch recipe).
If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use langwatch dataset get <slugOrId> to inspect it.
Apply the fix and re-run:

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

Step 5: Prevent the Recurrence

If the failure mode wasn't covered by any evaluator, add one (langwatch evaluator create) and wire it into the experiment.
If it only shows up in production, set up a monitor (langwatch monitor create) so online evaluation catches it before the next experiment does.

---name: eval-triagedescription: Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.license: MIT---# Triage Failing Experiments and EvaluationsFrom "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.## Prerequisites## Step 1: Find the Failing Run```bashlangwatch experiment list --limit 20 -o json # experiments in the projectlangwatch experiment list-runs <slug> -o json # runs for one experimentlangwatch experiment status <slug> -o json # latest run: status, progress, errorslangwatch experiment status <slug> --run-id <id> -o json```A run can fail two ways, and they triage differently:- **Execution failure**: the run errored out or stalled. `status` shows the error; jump to Step 4.- **Score regression**: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.## Step 2: Isolate the Failing Rows```bashlangwatch experiment results <slug> --filter failed -o jsonlangwatch experiment results <slug> --filter failed --evaluator <name> -o jsonlangwatch experiment results <slug> --run-id <id> --limit 50 -o json```- `--filter failed` keeps only the rows that failed at least one evaluator. Start there, not with the full result set.- `--evaluator <name>` shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.## Step 3: Inspect the Evaluators```bashlangwatch evaluator list -o jsonlangwatch evaluator get <idOrSlug> -o json```Before blaming the agent, rule out the scorer:- **LLM-judge evaluators**: check the model in `settings`. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.- **Thresholds**: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.- **Deterministic evaluators** (exact match, JSON validity): these don't drift; failures here are real.## Step 4: Root Cause1. Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)? `git log` on prompts and agent code usually answers this directly.2. If rows fail on retrieval or context: inspect a production trace of the same path (`langwatch trace search` / `langwatch trace get`; see the `debug-with-langwatch` recipe).3. If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use `langwatch dataset get <slugOrId>` to inspect it.4. Apply the fix and re-run:```bashlangwatch experiment run <slug> --waitlangwatch experiment status <slug> -o json```## Step 5: Prevent the Recurrence- If the failure mode wasn't covered by any evaluator, add one (`langwatch evaluator create`) and wire it into the experiment.- If it only shows up in production, set up a monitor (`langwatch monitor create`) so online evaluation catches it before the next experiment does.

Frequently asked questions