# Eval Triage

> Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.

- Skill: `langwatch/eval-triage` (Agent Skill)
- Install (CLI): `npx skillmds@latest add langwatch/eval-triage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/langwatch/eval-triage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: langwatch (https://skillmd.com/u/langwatch)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/langwatch/eval-triage

---


# Triage Failing Experiments and Evaluations

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

## Prerequisites

## Step 1: Find the Failing Run

```bash
langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json
```

A run can fail two ways, and they triage differently:

- **Execution failure**: the run errored out or stalled. `status` shows the error; jump to Step 4.
- **Score regression**: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

## Step 2: Isolate the Failing Rows

```bash
langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json
```

- `--filter failed` keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
- `--evaluator <name>` shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

## Step 3: Inspect the Evaluators

```bash
langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json
```

Before blaming the agent, rule out the scorer:

- **LLM-judge evaluators**: check the model in `settings`. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
- **Thresholds**: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
- **Deterministic evaluators** (exact match, JSON validity): these don't drift; failures here are real.

## Step 4: Root Cause

1. Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)? `git log` on prompts and agent code usually answers this directly.
2. If rows fail on retrieval or context: inspect a production trace of the same path (`langwatch trace search` / `langwatch trace get`; see the `debug-with-langwatch` recipe).
3. If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use `langwatch dataset get <slugOrId>` to inspect it.
4. Apply the fix and re-run:

```bash
langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json
```

## Step 5: Prevent the Recurrence

- If the failure mode wasn't covered by any evaluator, add one (`langwatch evaluator create`) and wire it into the experiment.
- If it only shows up in production, set up a monitor (`langwatch monitor create`) so online evaluation catches it before the next experiment does.