# Distill Test Set > Turn a large dataset (an existing oversized Coval test set, an export of past conversations, or a CSV/JSON of cases) into a small, high-signal Coval test set by removing duplicates, identifying unique scenarios, and selecting a representative, failure-weighted subset — then bulk-loading it with no row cap. Use when the user says "I have thousands of cases", "dedupe my test set", "my test set is too big", "turn this dataset into a test set", "pick representative scenarios", or "my CSV import only kept 10 / uploaded everything". - Skill: `coval-ai/distill-test-set` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add coval-ai/distill-test-set` - Raw SKILL.md: https://api.skillmd.com/api/skills/coval-ai/distill-test-set/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: coval-ai (https://skillmd.com/u/coval-ai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/coval-ai/distill-test-set --- # Distill Test Set Turn a large, noisy dataset into a small, high-signal Coval test set. Big datasets are the wrong input for simulation: running thousands of sims is slow, expensive, and likely to overwhelm the agent under test, and most rows are duplicates, paraphrases, or boilerplate. The goal is a **representative 10–50 case suite** that covers the agent's real scenarios once each, weighted toward known failures. Drive everything through the `coval` CLI and the bundled `scripts/distill.py`. Follow the phases in order. Present a summary and get confirmation before creating anything. If `$ARGUMENTS` is a Coval test set id (8 chars) or a file path, use it as the source in Phase 1. > **Why this skill instead of a plain import?** Coval has two loaders, and neither dedupes: > the CSV/XLSX importer and `coval test-cases create --stdin` load **every** row 1:1 > (great once you've chosen your rows), while "Create test set → attach a file" runs an > **LLM** that synthesizes only ~10–15 scenarios (a generation cap, not a representative > sample). This skill does the dedup + representative selection itself, then loads the > result with the uncapped path. ## Phase 0: Preflight + Inventory ### Step 1: Check authentication ```bash coval whoami ``` If not authenticated: `coval login` (get a key at https://app.coval.dev/settings → Organization → Manage → API Keys). No account? https://coval.dev. > **Before you start:** this skill needs only the `coval` CLI and `python3` (used by the > bundled script and a couple of small JSON reads) — no `jq` or other tools. If `coval` is a > shell alias on your machine, run the commands with `coval-cli` instead (`which coval` to check). ### Step 2: Inventory existing test sets (so we don't clobber one) ```bash coval test-sets list --format json ``` Note the ids/sizes. If the user wants to distill an existing Coval test set, confirm its id here. ## Phase 1: Locate + Profile the Source Dataset Determine where the large dataset lives. Two common cases: 1. **An existing, oversized Coval test set** — use `--source coval:` (the script pulls every case via the API, paginated). Confirm the id and its size: ```bash coval test-sets get --format json | python3 -c "import json,sys;d=json.load(sys.stdin);print(d.get('display_name'), d.get('test_case_count') or '(size shown in app)')" ``` 2. **A local file** — CSV, JSON, or NDJSON of past conversations / SME-reviewed rows / logs. Then identify the **field mapping** by inspecting a few rows: - the **user utterance / scenario** field (`--input-field`, default `input_str`) - the **expected answer / behavior** field (`--expected-field`, default `expected_behaviors`) - a **label** to stratify by — the intent or route or category (`--label-field`) - a **pass/fail** field if the data is review-graded (`--pass-fail-field`; values starting with `Pass`/`Fail`) If the source is a Coval test set whose metadata is packed into a pipe-delimited `description` (e.g. `journey=… | sub_topic=… | next_agent_status=… | sme_result=…`), add `--parse-coval-description` so those become usable labels. Load `references/distillation-method.md` if you need help choosing the field mapping or understanding what the script does. ## Phase 2: Distill Run the distiller. It (1) drops rows with no checkable answer, (2) removes exact + paraphrase duplicates, (3) caps boilerplate-convergence clusters (many different asks that all hit one generic fallback/greeting) to a small quota, and (4) selects a stratified, **failure-weighted** subset across your labels and languages. ```bash python3 scripts/distill.py \ --source \ --parse-coval-description \ # only if labels are in a pipe-delimited description --label-field \ --pass-fail-field \ --target-size 30 \ # start small: 10–50 --min-minority-ratio 0.3 \ # keep ~1/3 of a minority language (e.g. non-English); 0 to disable --out distilled.ndjson ``` The script prints a report to stderr — show it to the user: ``` dropped_low_info / boilerplate_groups / scenario_rows_after_dedup / selected / selected_fail / selected_cjk / selected_by_label ``` - No third-party packages are needed; paraphrase clustering uses a character n-gram heuristic by default. - For higher-quality clustering on multilingual data, set `OPENAI_API_KEY` and add `--embeddings on` (it falls back to the heuristic automatically if anything fails). - Tuning knobs: `--target-size`, `--fail-weight` (default 0.45), `--boilerplate-quota` (default 4), `--jaccard-threshold` / `--embed-threshold`. See `references/distillation-method.md`. ## Phase 3: Review + Adjust Show the user a sample of the distilled set and the label/pass-fail/language breakdown: ```bash python3 -c "import json;[print('IN :',json.loads(l)['input_str'][:70],'| EXP:',json.loads(l)['expected_output_str'][:60]) for l in open('distilled.ndjson')]" wc -l distilled.ndjson ``` Ask: "Does this coverage look right? (yes / re-run with different size or weighting / edit the file)" - Too many fallback/greeting rows → lower `--boilerplate-quota`. - Want more failure coverage → raise `--fail-weight` or `--target-size`. - Want to hand-curate → edit `distilled.ndjson` directly (one JSON object per line: `{"input_str", "expected_output_str", "description"}`). Keep every line valid JSON — the bulk loader stops at the first malformed line, so a stray typo can silently drop the rest. - Sanity-check coherence → confirm each `expected_output_str` actually answers its `input_str`. The distiller preserves whatever input/answer pairing your source had, so any misalignment already in the source data carries through (garbage in, garbage out). **Start small.** A 20–30 case suite you can run and trust beats a 2,000 case suite you can't. Get the agent passing these, then add a new test set for the next workflow. ## Phase 4: Create the Test Set + Bulk-Load (uncapped) Create a `SCENARIO` test set and load the distilled cases with `--stdin` — the **uncapped** path that loads exactly the rows you chose (no 10-case generation limit): ```bash TS=$(coval test-sets create --name " — Representative Suite" --type SCENARIO \ --description "Distilled from : deduped, failure-weighted representative scenarios" \ --format json | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])") cat distilled.ndjson | coval test-cases create --test-set-id "$TS" --stdin ``` > `--stdin` reads one JSON object per line (`input_str` required; `expected_output_str`, > `description` optional) and POSTs one test case per line with **no row cap**. This is the > fix for "the importer only kept ~10 cases". > **Note:** `SCENARIO` is the right type for persona-driven eval and is also the CLI default. > If an older CLI build rejects `--type`, just omit it (you still get SCENARIO). ## Phase 5: Verify + Next Steps ```bash coval test-cases list --test-set-id "$TS" --page-size 100 --format json | python3 -c "import json,sys;print(len(json.load(sys.stdin)))" ``` Confirm the count matches the distilled file, then show the app link: `https://app.coval.dev/test-sets/` ``` What's next? Create personas to drive these scenarios: /design-persona or /personas-from-artifacts Add expected behaviors / refine cases: /build-test-suite Configure evaluation metrics: /configure-metrics Launch an evaluation on the small set first: /quick-eval Iterate: once the agent passes this suite, distill a NEW test set for the next workflow rather than growing this one — narrow, branch-covering suites beat one giant set. ```