training-data

# training-data — the corpus a fine-tune actually eats You own the **training corpus**: the JSONL of chat turns, instruction triples, or preference pairs that a trainer reads. The deliverable is a validated, deduplicated, decontaminated, license-clean file in the **exact shape the trainer expects**, rendered through the **target model's chat template**. You stop the moment that file loads cleanly and round-trips through `apply_chat_template`. You do not choose LoRA rank or launch the run — that is [`finetuning`](../finetuning/SKILL.md) / [`unsloth`](../unsloth/SKILL.md). **Loud boundary.** This is *LLM training corpora* — messages, instruction triples, preference pairs. It is **not**: - Tabular row cleaning — nulls, dtypes, dedupe of CSV rows, category normalization → [`data-cleaning`](../data-cleaning/SKILL.md). - A *retrieval* corpus — chunking documents and embedding them for search → [`embeddings-search`](../embeddings-search/SKILL.md). - Actually training or serving — hyperparameters, the run, export → [`finetuning`](../finetuning/SKILL.md), [`unsloth`](../unsloth/SKILL.md), [`huggingface`](../huggingface/SKILL.md). **Version reality (verified July 2026 — re-verify, these move monthly).** TRL is on the **v1.x** line (its dataset-formats doc was tagged `v1.8.0` at author time); `transformers` is in the **4.57+** era (mixed text+vision data needs ≥4.57); `datasets` is **4.x** (the `Json()` feature type needs ≥4.7). Pin whatever you install — do not trust these numbers as current. ## 1. The format the trainer expects (pick by trainer, not by taste) The trainer dictates the columns. Get this wrong and TRL either errors or, worse, trains on a mangled string. Two axes: **format** (`standard` = plain strings vs `conversational` = `messages` lists) and **type** (the task). One JSON object per line = JSONL. | Trainer | Dataset type | Required keys | | --- | --- | --- | | `SFTTrainer` | language-modeling **or** prompt-completion | `messages` / `text`, or `prompt`+`completion` | | `DPOTrainer`, `ORPOTrainer`, `CPOTrainer` | preference (explicit prompt recommended) | `prompt`, `chosen`, `rejected` | | `KTOTrainer`, `BCOTrainer` | unpaired preference (binary label) | `prompt`, `completion`, `label` | | `RewardTrainer` | preference (implicit prompt) | `chosen`, `rejected` | | `GRPOTrainer`, `RLOOTrainer`, `PPOTrainer` | prompt-only | `prompt` | Tiny JSONL of each (conversational values are **lists of `{role, content}`**; `label` is a JSON boolean): ```jsonl # Alpaca instruction (standard) — classic; NOT a native TRL type, see below {"instruction": "Classify the sentiment.", "input": "The battery dies in an hour.", "output": "negative"} # Conversational messages (SFT) — the default for chat fine-tunes {"messages": [{"role": "system", "content": "You are a terse support agent."}, {"role": "user", "content": "My order never arrived."}, {"role": "assistant", "content": "Sorry about that — what is your order number?"}]} # Preference pair (DPO) — chosen beats rejected for the same prompt {"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "chosen": [{"role": "assistant", "content": "A hash map stores key-value pairs and finds a value by hashing its key to a bucket, giving average O(1) lookup."}], "rejected": [{"role": "assistant", "content": "It's a fast dictionary thing."}]} # KTO / unpaired preference — one completion + a good/bad boolean label {"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "completion": [{"role": "assistant", "content": "It's a fast dictionary thing."}], "label": false} ``` **Alpaca is not a native TRL type.** `{instruction, input, output}` is the Stanford-Alpaca convention, still common in Unsloth notebooks, but TRL trains on `text`/`messages`/`prompt`+ `completion`. You must either (a) map it into `messages` (instruction+input → user, output → assistant), or (b) render it into a single `text` string via a prompt template — **and append the EOS token yourself**, or the model never learns to stop (the #1 Unsloth-Alpaca bug). Prefer (a) `messages` for chat models. Full field matrix, tool-calling (`tools` column) and vision (`images`) extras, and every type→type conversion live in `references/formats.md`. ## 2. Chat templates — the silent run-wrecker A chat template is a **Jinja** string stored in the tokenizer (in `tokenizer_config.json` under `chat_template`, or a standalone `chat_template.jinja` in newer tokenizers). It maps a `messages` list to the exact token string the model was trained on — special tokens (`<|im_start|>`, `[INST]`, `<|start_header_id|>`, …) and all. You render it, you never hand-type it: ```python from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("<target-model>") # the model you will fine-tune # TRAINING: no trailing generation prompt — the assistant turn is already in the data text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False) # INFERENCE: add_generation_prompt=True appends the assistant turn-start so the model continues prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) ``` Three ways this silently destroys a run — **no error, just a worse model**: - **Hand-formatting the tokens.** Writing `<|im_start|>user\n…` strings yourself and getting one token, one newline, or the BOS wrong. Train-time string ≠ inference-time string → the model learns a distribution it is never served. Always render via `apply_chat_template`. - **Using the wrong model's template.** The template must be the one of the model you are fine-tuning. Copy Llama's template onto a Qwen fine-tune and every example is subtly malformed. - **A base model with no template at all.** Base (non-instruct) checkpoints often ship `chat_template = None`. `apply_chat_template` then raises — you must **choose and attach** a template (e.g. ChatML) and use that same one at inference forever after. Also decide **loss masking**: for chat SFT you usually train only on the assistant tokens (`completion_only_loss` / assistant-only masking in SFTTrainer, or a completion-only collator), so the model is not penalized for "predicting" the user's words. TRL applies the template for you when the dataset is conversational — let it, rather than pre-flattening to `text`. ## 3. Synthetic data & distillation Not enough real examples? Generate them. Two workhorses: **Self-Instruct** (seed a few hand-written examples, prompt a strong model to produce more, filter) and **Evol-Instruct** (iteratively mutate prompts to be harder/deeper). Wrap them in a pipeline framework rather than ad-hoc loops (see §7). **Licensing trap — read before you distill.** Generating your training data from *another model's outputs* ("distillation") is a **terms-of-service question, not just a quality one**. Some providers' terms restrict using their outputs to train competing models; some open-weight licenses carry naming/derivative obligations (e.g. Llama-derived data/models may inherit naming requirements). Never assert a model's license from memory — check the specific model card and provider ToS at author time (licenses change). If in doubt, distill from an openly-licensed-for-this-use model, and record the provenance per example. ## 4. Dedup + decontamination (skip these and your numbers lie) - **Dedup.** Exact dedup is trivial (hash the text). Real corpora need **near-dup** removal: **MinHash + LSH** (Jaccard similarity over shingles) catches templated/boilerplate repeats that inflate a few patterns. Dupes waste compute and bias the model toward whatever is over-represented. - **Decontamination — the one people forget.** Remove any training example that overlaps your **eval / benchmark test sets** (n-gram overlap, e.g. long-n-gram match against MMLU, GSM8K, your own held-out set). If test items leak into training, your eval score is **inflated** and meaningless — you measured memorization, not capability. Decontaminate *against every metric you will report*, including your private eval. Code for both in `references/synthesis-dedup-quality.md`. ## 5. Quality filtering — a few thousand clean beats a noisy dump **LIMA** (*Less Is More for Alignment*, arXiv **2305.11206**) is the anchor: ~1,000 carefully curated examples produced a strong instruction-follower — for **alignment/style** SFT, quality and diversity dominate raw volume. (This is about teaching *behavior/format*, not injecting a lot of new *knowledge* — a broad knowledge shift still wants scale.) Cheap, high-leverage filters, applied before you spend GPU hours: - **Length/format**: drop empty or truncated turns, runaway-length outliers, malformed JSON, wrong-role sequences (two `assistant` turns in a row, missing final assistant turn for SFT). - **Dedup + decontam** from §4. - **Diversity**: cluster/embed and prune near-identical intents so the set is not 80% one task. - **Model/heuristic scoring**: rate helpfulness/correctness (a reward model or an LLM judge) and keep the top slice — but audit the judge, LLM-as-judge has its own biases. ## 6. Licensing — two separate questions 1. **The dataset's own license** — what *you* release the JSONL under, and whether you *can* release it (aggregating others' data does not launder their licenses). 2. **Source-usage restrictions** — the terms on where each example *came from*: scraped-site ToS, the license of any base dataset you built on, and the model-output ToS from §3. These bind even if you never publish. Keep a provenance column so an audit can trace every row. State the license *class* and point at the source; never freeze a license as bare fact. ## 7. Tooling - **distilabel** (Argilla, now under Hugging Face) — the go-to synthetic-data / AI-feedback pipeline framework: composable `Step`/`Task` graphs (`TextGeneration`, `UltraFeedback`, `EvolInstruct`), serializable to YAML/JSON, outputs a `Distiset` you push to the Hub. v1.x. - **Argilla** — human-in-the-loop annotation/review UI to label and vet examples. - **HF `datasets`** — load/`map`/`filter`/`push_to_hub`; the substrate everything else speaks. - **Lilac** — dataset exploration/clustering for quality triage. **[verify — the open-source repo was archived (read-only) around July 2025 after the Databricks acquisition]**; treat as unmaintained OSS and confirm before depending on it. ## Worked lifecycle (build → validate → dedup → decontaminate → format → push) ```python from datasets import load_dataset from transformers import AutoTokenizer ds = load_dataset("json", data_files="raw.jsonl", split="train") tok = AutoTokenizer.from_pretrained("<target-model>") # 1. VALIDATE shape + render every row through the template (catches template errors NOW, # not after 3 GPU-hours). A base model with chat_template=None raises here — attach one. def render(ex): return {"text": tok.apply_chat_template(ex["messages"], tokenize=False, add_generation_prompt=False)} ds = ds.filter(lambda ex: isinstance(ex.get("messages"), list) and ex["messages"] and ex["messages"][-1]["role"] == "assistant") # SFT: must end on assistant ds = ds.map(render) # 2. DEDUP (near-dup) and 3. DECONTAMINATE against your eval set — see references for MinHash # + n-gram code; both are one filter pass each. # 4. PUSH with a data card recording license + provenance. ds.push_to_hub("me/support-sft", private=True) ``` Deep code — MinHash/LSH dedup, n-gram decontamination, a distilabel Self-Instruct pipeline, and the full conversion matrix — is in `references/`. ## Guardrails / gotchas - **Wrong format for the trainer** = hard error or silent garbage. Match the table in §1 to your trainer before generating a single row. - **Hand-typed chat tokens** train a distribution you never serve. Render via `apply_chat_template`. - **No EOS in Alpaca-`text` formatting** → the model never stops. Append it. - **Skipped decontamination** → inflated eval; you measured leakage. Non-negotiable. - **Distilling model outputs** can violate ToS. Provenance + license check first. - **Quantity worship** — a noisy 500k dump loses to a curated few-thousand for alignment (LIMA). - `label` in KTO/unpaired data is a JSON boolean (`true`/`false`), not the strings `"true"`/`"1"`. ## Related skills - [`finetuning`](../finetuning/SKILL.md) — consumes this corpus: chooses SFT vs DPO vs KTO, LoRA/QLoRA, hyperparameters, runs trl/peft. You hand it the file; it trains. - [`unsloth`](../unsloth/SKILL.md) — one fast single-GPU training backend + GGUF export; its notebooks expect exactly the Alpaca/`messages` shapes you produce here. - [`huggingface`](../huggingface/SKILL.md) — the Hub you `push_to_hub` the dataset to, model cards, and hosted/routed inference of the *result*. - [`data-cleaning`](../data-cleaning/SKILL.md) — upstream when your raw source is dirty *tabular* rows; it hands you clean rows, you turn rows into training examples. ## Checklist - [ ] Format matches the target **trainer** (§1 table); one JSON object per line. - [ ] Every row renders through the **target model's** `apply_chat_template` without error. - [ ] SFT rows end on an `assistant` turn; preference rows have distinct `chosen`/`rejected`; KTO `label` is a boolean. - [ ] Loss masking / EOS handling decided (assistant-only loss; EOS appended if flattening to `text`). - [ ] Near-duplicates removed (MinHash/LSH); exact dupes gone. - [ ] **Decontaminated** against every eval/benchmark you will report. - [ ] Quality-filtered (length/format/diversity/score) — curated over bulk. - [ ] Dataset license set **and** source-usage/model-output ToS checked; provenance recorded. - [ ] Data card written; pushed (private first).