trl-training

name	trl-training
description	Trains Hugging Face TRL jobs: trl sft/dpo/grpo/kto/rloo/reward, YAML configs, LoRA/PEFT, and Accelerate multi-GPU. Use when fine-tuning or aligning transformer LMs with TRL CLI. Not for vLLM OpenAI serving, local Hub evals (hugging-face-community-evals), or sentence-transformers embedding training.
version	1.0.1
risk	unknown
source	https://github.com/huggingface/skills/tree/main/skills/trl-training
source_repo	huggingface/skills
source_type	official
date_added	2026-07-01T00:00:00.000Z
license	Apache-2.0
license_source	https://github.com/huggingface/skills/blob/main/LICENSE

TRL Training

When to Use

Use this skill when the user needs to train or fine-tune transformer language models using the TRL (Transformers Reinforcement Learning) library. Trigger keywords and scenarios:

• SFT — supervised fine-tuning on instruction-following or conversational datasets
• DPO — direct preference optimization with chosen/rejected pairs
• GRPO — group relative policy optimization with reward functions or LLM-as-a-judge
• KTO — Kahneman-Tversky optimization (use trl kto CLI)
• RLOO — REINFORCE Leave-One-Out online RL with generation-based rewards
• Reward Model — training a reward model for RLHF
• Configuration via YAML, distributed training with Accelerate, LoRA/PEFT adapters, or troubleshooting TRL training runs

You are an expert at using the TRL library to train and fine-tune large language models.

Prerequisites

1. Install TRL (requires Python ≥ 3.9, PyTorch, and Hugging Face Transformers):

   pip install trl
   

2. Authenticate with Hugging Face Hub (needed for gated models, pushing to hub, or private datasets):

   hf auth login
   

   Use a token placeholder such as YOUR_HF_TOKEN — never commit real tokens.

3. Verify GPU availability (training expects CUDA; CPU-only is impractical for all but tiny models):

   python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"
   

4. Windows host notes (PowerShell primary):

   • Multi-line backslash-continuation commands shown below are bash syntax. In PowerShell, use backtick (`) as the line-continuation character, or run the command as a single line.
   • For YAML config files, use forward slashes or escaped backslashes in paths.
   • Accelerate config default path on Windows: ~/.cache/huggingface/accelerate/default_config.yaml

Procedure

1. Choose the training method

Goal	Command	Typical dataset schema