# Compare Trials

> Analyze graded `trial_result` JSONL outputs from the eval pipeline, including reliability and custom slicing.

- Skill: `plaited/compare-trials` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add plaited/compare-trials`
- Raw SKILL.md: https://api.skillmd.com/api/skills/plaited/compare-trials/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: ISC
- Author: plaited (https://skillmd.com/u/plaited)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/plaited/compare-trials

---


# Compare Trials

Use this skill for analysis over graded JSONL outputs, typically after:

```bash
agent-eval-harness eval '{"mode":"grade",...}' > graded.jsonl
```

## Default path

For normal baseline-vs-challenger comparisons, use:

```bash
agent-eval-harness eval '{"mode":"compare",...}'
```

Use custom scripts only when suite-specific analysis is required.

## Custom analysis use cases

- cost analysis
- token usage analysis
- slices by task metadata (category, difficulty, source)
- custom regression gates
- per-agent metadata summaries

## Script guidance

When writing a custom script:

1. Parse JSONL rows as `trial_result`.
2. Validate graded rows (`pass` and `score` non-null).
3. Group by stable identity fields (`taskId`, `trialIndex`, `runId`).
4. Separate:
   - standalone run metrics
   - comparable-overlap metrics
5. Report uncertainty where possible (bootstrap confidence intervals).
6. Use `exactPassAtK` naming for combinatorics-based pass@k metrics (do not label exact values as `estimatedPassAtK`).

Reference helpers remain in `references/` for bootstrap/statistics utilities.