Trace To Training Data

Trace To Training Data

This skill assumes eval-harness-first already graded the traces being converted here — goldens, graders, and runs/<run-id>/results.json all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.

Input: graded traces — eval/goldens.jsonl plus runs/<run-id>/results.json, each row carrying a task_id, a verdict from the grader, and a reward when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):

{"task_id": "t-042", "trace_id": "t-042-a3",
 "messages": [{"role": "user", "content": "..."}],
 "verdict": "pass", "reward": 0.91,
 "grader": "exact_match"}

Output format: rows shaped exactly like dataset-curation's Format Selection table — SFT messages rows or DPO prompt/chosen/rejected pairs — so this skill's output is that skill's input with no reshaping step in between.

The Principle

The eval harness already did the labeling work: every trace in results.json carries a verdict, and often a reward, before this skill ever touches it. Converting a graded trace into a training row is mechanical — pick a shape from dataset-curation's table, map fields, write JSONL. Curation is the work that remains — which traces clear a quality bar, which pairs are informative, and which rows must never enter the training set at all.

Treat any conversion step that requires re-judging a trace as a sign the harness is missing a grader, not a gap this skill should paper over. A trace with no verdict or reward isn't convertible yet — route it back to eval-harness-first first, don't hand-label it here to unblock conversion.

SFT From Traces

• Keep the top-reward fraction of successful trajectories, not every passing one. Rank passing traces by reward and take a fraction (the Agent-lightning pattern) rather than every trace that merely cleared the pass bar — a trace that barely passed is a weaker SFT signal than one that scored well above threshold.
• Expert-corrected failures become gold SFT examples directly (the Langfuse pattern) — when a human edits a failing trace's output into a correct one, that correction needs no reward threshold; a human already validated it. Route corrections straight into the SFT set.
• Step-level masking beats whole-trajectory discard for multi-step traces. When only some steps in a multi-step trajectory are bad, mask the loss on the bad steps and keep the good ones, rather than discarding the whole trajectory. SRFT reports 32.2% vs. 30.9% on SWE-bench for step-level critic masking over trajectory discard — a real, if modest, gap from the finer-grained cut.

Preference Pairs From Traces

• Build pairs from passing-vs-failing trajectories on the SAME task, never from unrelated best- and worst-scoring traces pulled across different tasks — cross-task pairs teach the model to prefer one task over another, not one response over another.
• Select the rejected member at μ−2σ of the reward distribution for that task, never the absolute minimum. preference-optimization's Pair Construction section owns the full selection formula; this skill supplies the graded trajectories it consumes.
• Judge-scored delta selection cuts pair volume without cutting signal. Score each candidate pair by chosen-minus-rejected judge delta and keep only the highest-delta subset — the top 5k of a 16.5k candidate pool matched the full pool's downstream result. Build the full candidate set first, then filter by delta; don't cap generation at 5k up front.

Hygiene

• Scan for secrets and PII before any row ships, and redact what's found. Traces sourced from production logs can carry credentials, API keys, tokens, or customer data — run a secret/PII scan over every SFT and DPO row and redact matches; conversion fails closed (the row is dropped, not shipped with the raw content) if sensitive fields remain after redaction. Never commit secrets.
• Eval goldens must never leak into training data. Hold every eval/goldens.jsonl ID out of every converted SFT and DPO set — a trace that also appears as a golden trains on the exact item the checkpoint gets graded against later, silently inflating every subsequent eval run.
• Dedup against the training set, not just within the newly converted rows — exact-match or embedding-similarity, matching dataset-curation's dedup method field, run against whatever training data already exists before this batch merges in.
• Provenance goes into the dataset card. Every converted row must trace back to its source run_id and trace_id — dataset-curation's Provenance field checks for exactly this link back to trace-to-training-data output; a row with no traceable source isn't ready to merge.

Related Skills

• eval-harness-first — produces the graded traces this skill converts; a trace with no verdict or reward isn't convertible yet, route it back there before conversion.
• dataset-curation — owns the target formats and the dataset card this skill's provenance data feeds; converted rows must match its Format Selection table field names exactly, not an approximation of them.
• preference-optimization — consumes the DPO pairs this skill builds and owns the full μ−2σ rejection-selection formula referenced above.

Worked JSONL-to-JSONL conversions — graded trace to SFT row, trace pair to DPO pair, correction to SFT row, the rejection-sampling loop, and the goldens-holdout check — live in references/conversion-recipes.md.

Source: wshobson/agents → plugins/llm-finetuning/skills/trace-to-training-data/SKILL.md