Training Data

training-data — the corpus a fine-tune actually eats

You own the training corpus: the JSONL of chat turns, instruction triples, or preference pairs that a trainer reads. The deliverable is a validated, deduplicated, decontaminated, license-clean file in the exact shape the trainer expects, rendered through the target model's chat template. You stop the moment that file loads cleanly and round-trips through apply_chat_template. You do not choose LoRA rank or launch the run — that is finetuning / unsloth.

Loud boundary. This is LLM training corpora — messages, instruction triples, preference pairs. It is not:

• Tabular row cleaning — nulls, dtypes, dedupe of CSV rows, category normalization → data-cleaning.
• A retrieval corpus — chunking documents and embedding them for search → embeddings-search.
• Actually training or serving — hyperparameters, the run, export → finetuning, unsloth, huggingface.

Version reality (verified July 2026 — re-verify, these move monthly). TRL is on the v1.x line (its dataset-formats doc was tagged v1.8.0 at author time); transformers is in the 4.57+ era (mixed text+vision data needs ≥4.57); datasets is 4.x (the Json() feature type needs ≥4.7). Pin whatever you install — do not trust these numbers as current.

1. The format the trainer expects (pick by trainer, not by taste)

The trainer dictates the columns. Get this wrong and TRL either errors or, worse, trains on a mangled string. Two axes: format (standard = plain strings vs conversational = messages lists) and type (the task). One JSON object per line = JSONL.

Tiny JSONL of each (conversational values are lists of {role, content}; label is a JSON boolean):

# Alpaca instruction (standard) — classic; NOT a native TRL type, see below
{"instruction": "Classify the sentiment.", "input": "The battery dies in an hour.", "output": "negative"}

# Conversational messages (SFT) — the default for chat fine-tunes
{"messages": [{"role": "system", "content": "You are a terse support agent."}, {"role": "user", "content": "My order never arrived."}, {"role": "assistant", "content": "Sorry about that — what is your order number?"}]}

# Preference pair (DPO) — chosen beats rejected for the same prompt
{"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "chosen": [{"role": "assistant", "content": "A hash map stores key-value pairs and finds a value by hashing its key to a bucket, giving average O(1) lookup."}], "rejected": [{"role": "assistant", "content": "It's a fast dictionary thing."}]}

# KTO / unpaired preference — one completion + a good/bad boolean label
{"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "completion": [{"role": "assistant", "content": "It's a fast dictionary thing."}], "label": false}

Alpaca is not a native TRL type. {instruction, input, output} is the Stanford-Alpaca convention, still common in Unsloth notebooks, but TRL trains on text/messages/prompt+ completion. You must either (a) map it into messages (instruction+input → user, output → assistant), or (b) render it into a single text string via a prompt template — and append the EOS token yourself, or the model never learns to stop (the #1 Unsloth-Alpaca bug). Prefer (a) messages for chat models. Full field matrix, tool-calling (tools column) and vision (images) extras, and every type→type conversion live in references/formats.md.

2. Chat templates — the silent run-wrecker

A chat template is a Jinja string stored in the tokenizer (in tokenizer_config.json under chat_template, or a standalone chat_template.jinja in newer tokenizers). It maps a messages list to the exact token string the model was trained on — special tokens (<|im_start|>, [INST], <|start_header_id|>, …) and all. You render it, you never hand-type it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<target-model>")   # the model you will fine-tune

# TRAINING: no trailing generation prompt — the assistant turn is already in the data
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)

# INFERENCE: add_generation_prompt=True appends the assistant turn-start so the model continues
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)

Three ways this silently destroys a run — no error, just a worse model:

• Hand-formatting the tokens. Writing <|im_start|>user\n… strings yourself and getting one token, one newline, or the BOS wrong. Train-time string ≠ inference-time string → the model learns a distribution it is never served. Always render via apply_chat_template.
• Using the wrong model's template. The template must be the one of the model you are fine-tuning. Copy Llama's template onto a Qwen fine-tune and every example is subtly malformed.
• A base model with no template at all. Base (non-instruct) checkpoints often ship chat_template = None. apply_chat_template then raises — you must choose and attach a template (e.g. ChatML) and use that same one at inference forever after.

Also decide loss masking: for chat SFT you usually train only on the assistant tokens (completion_only_loss / assistant-only masking in SFTTrainer, or a completion-only collator), so the model is not penalized for "predicting" the user's words. TRL applies the template for you when the dataset is conversational — let it, rather than pre-flattening to text.

3. Synthetic data & distillation

Not enough real examples? Generate them. Two workhorses: Self-Instruct (seed a few hand-written examples, prompt a strong model to produce more, filter) and Evol-Instruct (iteratively mutate prompts to be harder/deeper). Wrap them in a pipeline framework rather than ad-hoc loops (see §7).

Licensing trap — read before you distill. Generating your training data from another model's outputs ("distillation") is a terms-of-service question, not just a quality one. Some providers' terms restrict using their outputs to train competing models; some open-weight licenses carry naming/derivative obligations (e.g. Llama-derived data/models may inherit naming requirements). Never assert a model's license from memory — check the specific model card and provider ToS at author time (licenses change). If in doubt, distill from an openly-licensed-for-this-use model, and record the provenance per example.

4. Dedup + decontamination (skip these and your numbers lie)

• Dedup. Exact dedup is trivial (hash the text). Real corpora need near-dup removal: MinHash + LSH (Jaccard similarity over shingles) catches templated/boilerplate repeats that inflate a few patterns. Dupes waste compute and bias the model toward whatever is over-represented.
• Decontamination — the one people forget. Remove any training example that overlaps your eval / benchmark test sets (n-gram overlap, e.g. long-n-gram match against MMLU, GSM8K, your own held-out set). If test items leak into training, your eval score is inflated and meaningless — you measured memorization, not capability. Decontaminate against every metric you will report, including your private eval. Code for both in references/synthesis-dedup-quality.md.

5. Quality filtering — a few thousand clean beats a noisy dump

LIMA (Less Is More for Alignment, arXiv 2305.11206) is the anchor: ~1,000 carefully curated examples produced a strong instruction-follower — for alignment/style SFT, quality and diversity dominate raw volume. (This is about teaching behavior/format, not injecting a lot of new knowledge — a broad knowledge shift still wants scale.) Cheap, high-leverage filters, applied before you spend GPU hours:

• Length/format: drop empty or truncated turns, runaway-length outliers, malformed JSON, wrong-role sequences (two assistant turns in a row, missing final assistant turn for SFT).
• Dedup + decontam from §4.
• Diversity: cluster/embed and prune near-identical intents so the set is not 80% one task.
• Model/heuristic scoring: rate helpfulness/correctness (a reward model or an LLM judge) and keep the top slice — but audit the judge, LLM-as-judge has its own biases.

6. Licensing — two separate questions

1. The dataset's own license — what you release the JSONL under, and whether you can release it (aggregating others' data does not launder their licenses).
2. Source-usage restrictions — the terms on where each example came from: scraped-site ToS, the license of any base dataset you built on, and the model-output ToS from §3. These bind even if you never publish. Keep a provenance column so an audit can trace every row.

State the license class and point at the source; never freeze a license as bare fact.

7. Tooling

• distilabel (Argilla, now under Hugging Face) — the go-to synthetic-data / AI-feedback pipeline framework: composable Step/Task graphs (TextGeneration, UltraFeedback, EvolInstruct), serializable to YAML/JSON, outputs a Distiset you push to the Hub. v1.x.
• Argilla — human-in-the-loop annotation/review UI to label and vet examples.
• HF datasets — load/map/filter/push_to_hub; the substrate everything else speaks.
• Lilac — dataset exploration/clustering for quality triage. [verify — the open-source repo was archived (read-only) around July 2025 after the Databricks acquisition]; treat as unmaintained OSS and confirm before depending on it.

Worked lifecycle (build → validate → dedup → decontaminate → format → push)

from datasets import load_dataset
from transformers import AutoTokenizer

ds  = load_dataset("json", data_files="raw.jsonl", split="train")
tok = AutoTokenizer.from_pretrained("<target-model>")

# 1. VALIDATE shape + render every row through the template (catches template errors NOW,
#    not after 3 GPU-hours). A base model with chat_template=None raises here — attach one.
def render(ex):
    return {"text": tok.apply_chat_template(ex["messages"], tokenize=False,
                                            add_generation_prompt=False)}
ds = ds.filter(lambda ex: isinstance(ex.get("messages"), list) and ex["messages"]
               and ex["messages"][-1]["role"] == "assistant")   # SFT: must end on assistant
ds = ds.map(render)

# 2. DEDUP (near-dup) and 3. DECONTAMINATE against your eval set — see references for MinHash
#    + n-gram code; both are one filter pass each.

# 4. PUSH with a data card recording license + provenance.
ds.push_to_hub("me/support-sft", private=True)

Deep code — MinHash/LSH dedup, n-gram decontamination, a distilabel Self-Instruct pipeline, and the full conversion matrix — is in references/.

Guardrails / gotchas

• Wrong format for the trainer = hard error or silent garbage. Match the table in §1 to your trainer before generating a single row.
• Hand-typed chat tokens train a distribution you never serve. Render via apply_chat_template.
• No EOS in Alpaca-text formatting → the model never stops. Append it.
• Skipped decontamination → inflated eval; you measured leakage. Non-negotiable.
• Distilling model outputs can violate ToS. Provenance + license check first.
• Quantity worship — a noisy 500k dump loses to a curated few-thousand for alignment (LIMA).
• label in KTO/unpaired data is a JSON boolean (true/false), not the strings "true"/"1".

Related skills

• finetuning — consumes this corpus: chooses SFT vs DPO vs KTO, LoRA/QLoRA, hyperparameters, runs trl/peft. You hand it the file; it trains.
• unsloth — one fast single-GPU training backend + GGUF export; its notebooks expect exactly the Alpaca/messages shapes you produce here.
• huggingface — the Hub you push_to_hub the dataset to, model cards, and hosted/routed inference of the result.
• data-cleaning — upstream when your raw source is dirty tabular rows; it hands you clean rows, you turn rows into training examples.

Checklist

• Format matches the target trainer (§1 table); one JSON object per line.
• Every row renders through the target model's apply_chat_template without error.
• SFT rows end on an assistant turn; preference rows have distinct chosen/rejected; KTO label is a boolean.
• Loss masking / EOS handling decided (assistant-only loss; EOS appended if flattening to text).
• Near-duplicates removed (MinHash/LSH); exact dupes gone.
• Decontaminated against every eval/benchmark you will report.
• Quality-filtered (length/format/diversity/score) — curated over bulk.
• Dataset license set and source-usage/model-output ToS checked; provenance recorded.
• Data card written; pushed (private first).