Eval Triage

Triage Failing Experiments and Evaluations

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

Prerequisites

Use langwatch docs <path> to read documentation as Markdown. Some useful entry points:

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with langwatch --help and langwatch <subcommand> --help. List and get commands accept --format json for machine-readable output. Every list command takes --limit <n> to cap the rows and --jq <expr> to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (--jq '.traces | length'), and read how many there are in all at .pagination.total. Bare --jq length counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append .md to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run npx --yes langwatch report --user-approved with a --title and --summary (or --session <transcript.jsonl>) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without --user-approved, and --dry-run prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with --email. With --session, always run --dry-run first and let the user read the payload, because a transcript carries content they never reviewed. npx --yes langwatch report --help explains the options.

Step 1: Find the Failing Run

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

A run can fail two ways, and they triage differently:

• Execution failure: the run errored out or stalled. status shows the error; jump to Step 4.
• Score regression: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

Step 2: Isolate the Failing Rows

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

• --filter failed keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
• --evaluator <name> shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

Step 3: Inspect the Evaluators

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

Before blaming the agent, rule out the scorer:

• LLM-judge evaluators: check the model in settings. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
• Thresholds: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
• Deterministic evaluators (exact match, JSON validity): these don't drift; failures here are real.

Step 4: Root Cause

1. Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)? git log on prompts and agent code usually answers this directly.
2. If rows fail on retrieval or context: inspect a production trace of the same path (langwatch trace search / langwatch trace get; see the debug-with-langwatch recipe).
3. If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use langwatch dataset get <slugOrId> to inspect it.
4. Apply the fix and re-run:

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

Step 5: Prevent the Recurrence

• If the failure mode wasn't covered by any evaluator, add one (langwatch evaluator create) and wire it into the experiment.
• If it only shows up in production, set up a monitor (langwatch monitor create) so online evaluation catches it before the next experiment does.