trl

# TRL Library Reference ## Package Structure TRL is organized around a trainer hierarchy that extends Hugging Face `transformers.Trainer`. ``` trl/ ├── trainer/ │ ├── grpo_trainer.py # GRPOTrainer │ ├── grpo_config.py # GRPOConfig │ ├── sft_trainer.py # SFTTrainer (supervised fine-tuning) │ ├── dpo_trainer.py # DPOTrainer (direct preference optimization) │ ├── kto_trainer.py # KTOTrainer (Kahneman-Tversky optimization) │ ├── online_dpo_trainer.py # OnlineDPOTrainer │ ├── utils.py # Shared utilities (log probs, decoding, padding) │ └── ... ├── models/ │ └── modeling_value_head.py # Value head for PPO-style trainers ├── data_utils.py ├── commands/ # CLI entry points └── ... ``` ## Trainer Hierarchy All TRL trainers extend `transformers.Trainer`: ``` transformers.Trainer ├── SFTTrainer # Supervised fine-tuning ├── DPOTrainer # Direct preference optimization ├── GRPOTrainer # Group relative policy optimization ├── KTOTrainer # Kahneman-Tversky optimization └── OnlineDPOTrainer # Online DPO ``` Each trainer overrides `compute_loss` with its specific objective, and RL-based trainers (GRPO, OnlineDPO) additionally override `training_step` to add a generation phase before the optimization step. ## Shared Utility Functions (trainer/utils.py) These utilities are used across multiple trainers. Read the source before modifying; the contracts below are what callers rely on. ### `selective_log_softmax(logits, index)` Memory-efficient per-token log-probability. Equivalent in value to `F.log_softmax(logits, dim=-1).gather(...)` at the selected token positions, but avoids materializing the full vocab-sized tensor. **Contract:** - Input: `logits [B, T, V]`, `index [B, T]` - Output: `log_probs [B, T]`, each entry a valid log-probability (i.e. non-positive) - Must agree with `F.log_softmax` to within numerical tolerance on the same inputs ### `decode_and_strip_padding(input_ids, tokenizer)` Converts a batch of token ID tensors into the cleaned text strings that the reward function will score. **Contract:** - Input: `input_ids [B, T]`, tokenizer - Output: `list[str]` of length `B` - Strips padding and decoder artefacts - Handles any reasoning-block conventions the library supports; the exact policy for complete, incomplete, and absent reasoning markers is defined in the implementation ### Other Utilities - `pad` / `pad_to_length` — Pad tensors to equal or specific lengths - Various tokenizer helpers for batch processing ## Configuration System All TRL configs extend `transformers.TrainingArguments`. Each trainer adds its own fields: | Config | Trainer | Key fields | |--------|---------|------------| | `SFTConfig` | SFTTrainer | `max_seq_length`, `packing`, `dataset_text_field` | | `DPOConfig` | DPOTrainer | `beta`, `loss_type`, `reference_free` | | `GRPOConfig` | GRPOTrainer | `num_generations`, `beta`, `epsilon`, `reward_functions` | | `KTOConfig` | KTOTrainer | `beta`, `desirable_weight`, `undesirable_weight` | ## Available References | File | Contents | When to load | |------|----------|-------------| | `references/trl-codebase.md` | Module-by-module guide to TRL source: detailed breakdown of each trainer, model wrappers, data utilities, and CLI commands | When navigating unfamiliar parts of TRL beyond the trainer layer, or when you need details about a specific non-GRPO trainer |