Skip to main content ホーム クリエイター kayforkind skill-slice trl-training
trl-training Trains Hugging Face TRL jobs: trl sft/dpo/grpo/kto/rloo/reward, YAML configs, LoRA/PEFT, and Accelerate multi-GPU. Use when fine-tuning or aligning transformer LMs with TRL CLI. Not for vLLM OpenAI serving, local Hub evals (hugging-face-community-evals), or sentence-transformers embedding training.
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/Kayforkind/skill-slice --skill trl-trainingコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills Open a creative mind on the current context and invent a leap the user did not know to ask for — a feature, protocol, CLI, demo, architecture, product move, prose, experiment, or visual. Use when they run /awe-me or /inspire-me, say “awe me” or “inspire me”, want surprise, adjacent possible, make-strange, or are stuck recognizing instead of seeing. Not a brainstorm list. Not /better (quality pass). Not a feasibility grill.
ab-testing-design-and-analysis Design, power, run, and analyze statistically valid A/B/N product experiments (sample size, duration, SRM, sequential monitoring, multiple comparisons). Use when pre-registering a test or reading out a controlled split. Not for event instrumentation, funnel diagnosis, wandb ML tracking (weights-and-biases), MVP smoke tests (mvp-scoping-and-risk-test), or RICE scoring (feature-prioritization-frameworks).
Runs Astropy BoxLeastSquares (BLS) on photometric time series: autopower/power grids, depth SNR, odd/even vetting. Use for periodic box-shaped transit dips or eclipsing binaries (Kovács 2002). Not for Transit Least Squares limb-darkened templates, Lomb-Scargle sinusoids, or non-photometric series. API: astropy.timeseries.BoxLeastSquares (Astropy 8).
name trl-training description Trains Hugging Face TRL jobs: trl sft/dpo/grpo/kto/rloo/reward, YAML configs, LoRA/PEFT, and Accelerate multi-GPU. Use when fine-tuning or aligning transformer LMs with TRL CLI. Not for vLLM OpenAI serving, local Hub evals (hugging-face-community-evals), or sentence-transformers embedding training. version 1.0.1 risk unknown source https://github.com/huggingface/skills/tree/main/skills/trl-training source_repo huggingface/skills source_type official date_added 2026-07-01T00:00:00.000Z license Apache-2.0 license_source https://github.com/huggingface/skills/blob/main/LICENSE
TRL Training
When to Use
Use this skill when the user needs to train or fine-tune transformer language models using the TRL (Transformers Reinforcement Learning) library. Trigger keywords and scenarios:
SFT — supervised fine-tuning on instruction-following or conversational datasetsDPO — direct preference optimization with chosen/rejected pairsGRPO — group relative policy optimization with reward functions or LLM-as-a-judgeKTO — Kahneman-Tversky optimization (use trl kto CLI)RLOO — REINFORCE Leave-One-Out online RL with generation-based rewardsReward Model — training a reward model for RLHFConfiguration via YAML, distributed training with Accelerate, LoRA/PEFT adapters, or troubleshooting TRL training runs
You are an expert at using the TRL library to train and fine-tune large language models.
Prerequisites
Install TRL (requires Python ≥ 3.9, PyTorch, and Hugging Face Transformers):
pip install trl
Authenticate with Hugging Face Hub (needed for gated models, pushing to hub, or private datasets):
hf auth login
Use a token placeholder such as YOUR_HF_TOKEN — never commit real tokens.
Verify GPU availability (training expects CUDA; CPU-only is impractical for all but tiny models):
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"
Windows host notes (PowerShell primary):
Multi-line backslash-continuation commands shown below are bash syntax. In PowerShell, use backtick (`) as the line-continuation character, or run the command as a single line.
For YAML config files, use forward slashes or escaped backslashes in paths.
Accelerate config default path on Windows: ~/.cache/huggingface/accelerate/default_config.yaml
Procedure
1. Choose the training method
Goal Command Typical dataset schema
Instruction / chat fine-tuning trl sftprompt + completion or conversational
Preference alignment trl dpoprompt, chosen, rejected
RL with reward functions trl grpoprompt (+ reward function)
Online RL with reward model trl rlooprompt (+ reward model)
Train a reward model trl rewardchosen, rejected
Best practice: Always SFT a base model before preference alignment (DPO/GRPO/RLOO).
2. Supervised Fine-Tuning (SFT) trl sft \
--model_name_or_path Qwen/Qwen2-0.5B \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--output_dir Qwen2-0.5B-SFT \
--push_to_hub
LoRA adapter training (lower memory, faster):
trl sft \
--model_name_or_path Qwen/Qwen2-0.5B \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-4 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--use_peft \
--lora_r 32 \
--lora_alpha 16 \
--output_dir Qwen2-0.5B-SFT \
--push_to_hub
3. Direct Preference Optimization (DPO) trl dpo \
--dataset_name trl-lib/ultrafeedback_binarized \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--learning_rate 5.0e-7 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--max_steps 1000 \
--gradient_accumulation_steps 8 \
--eval_strategy steps \
--eval_steps 50 \
--output_dir Qwen2-0.5B-DPO \
--no_remove_unused_columns
trl dpo \
--dataset_name trl-lib/ultrafeedback_binarized \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--learning_rate 5.0e-6 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--max_steps 1000 \
--gradient_accumulation_steps 8 \
--eval_strategy steps \
--eval_steps 50 \
--output_dir Qwen2-0.5B-DPO \
--no_remove_unused_columns \
--use_peft \
--lora_r 32 \
--lora_alpha 16
4. Group Relative Policy Optimization (GRPO) trl grpo \
--model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/gsm8k \
--reward_funcs accuracy_reward \
--output_dir Qwen2-0.5B-GRPO \
--push_to_hub
5. REINFORCE Leave-One-Out (RLOO) trl rloo \
--model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/tldr \
--reward_model_name_or_path sentiment-analysis:nlptown/bert-base-multilingual-uncased-sentiment \
--output_dir Qwen2-0.5B-RLOO \
--push_to_hub
6. Reward Model Training trl reward \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized \
--output_dir Qwen2-0.5B-Reward \
--per_device_train_batch_size 8 \
--num_train_epochs 1 \
--learning_rate 1.0e-5 \
--eval_strategy steps \
--eval_steps 50 \
--max_length 2048
trl reward \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized \
--output_dir Qwen2-0.5B-Reward-LoRA \
--per_device_train_batch_size 8 \
--num_train_epochs 1 \
--learning_rate 1.0e-4 \
--eval_strategy steps \
--eval_steps 50 \
--max_length 2048 \
--use_peft \
--lora_task_type SEQ_CLS \
--lora_r 32 \
--lora_alpha 16
7. Configuration Files (YAML) All CLI arguments can be placed in a YAML config for reproducibility.
model_name_or_path: Qwen/Qwen2.5-0.5B
dataset_name: trl-lib/Capybara
learning_rate: 2.0e-5
num_train_epochs: 1
per_device_train_batch_size: 8
gradient_accumulation_steps: 2
output_dir: ./sft_output
use_peft: true
lora_r: 16
lora_alpha: 16
report_to: trackio
trl sft --config sft_config.yaml
Override individual values:
trl sft --config sft_config.yaml --learning_rate 1.0e-5
8. Distributed Training (Accelerate) TRL integrates with Accelerate for multi-GPU and multi-node training.
Multi-GPU via --num_processes:
trl sft --config sft_config.yaml --num_processes 4
Predefined Accelerate configs (provided by TRL): single_gpu, multi_gpu, fsdp1, fsdp2, zero1, zero2, zero3
trl sft --config sft_config.yaml --accelerate_config zero2
trl sft --config sft_config.yaml --accelerate_config fsdp2
trl sft --config sft_config.yaml --accelerate_config zero3
Custom Accelerate config:
accelerate config
trl sft --config sft_config.yaml --config_file ~/.cache/huggingface/accelerate/default_config.yaml
9. Monitoring Add --report_to to any command for experiment tracking:
--report_to trackio (TRL's built-in)--report_to wandb (Weights & Biases — requires WANDB_API_KEY=YOUR_KEY)--report_to tensorboard
Pitfalls
CUDA Out of Memory
Reduce --per_device_train_batch_size and increase --gradient_accumulation_steps to keep effective batch size constant.
Enable --use_peft for LoRA training (dramatically lower VRAM).
Enable --gradient_checkpointing to trade compute for memory.
Try a smaller model or shorter sequence truncation.
Dataset Loading Issues
Verify the dataset exists on the Hugging Face Hub or at the local path provided.
Check that dataset columns match the expected schema for the chosen method (see table in step 1).
Use --dataset_config for multi-config datasets.
Inspect before training:
from datasets import load_dataset
ds = load_dataset("trl-lib/Capybara" )
print (ds)
Model Loading Issues
Verify the model exists on the Hugging Face Hub.
Gated models require authentication: hf auth login.
For local models, provide an absolute path.
Ensure sufficient disk space and system RAM for model weights.
Slow Training
Enable --packing for short-sequence datasets (reduces padding waste).
Increase --per_device_train_batch_size if VRAM allows.
Enable --tf32 on Ampere GPUs (A100, RTX 30xx+).
Enable --bf16 on supported hardware.
Use multi-GPU via --num_processes or an Accelerate config.
Generation Issues (GRPO / RLOO)
Check prompt format in the dataset matches what the model expects.
Adjust --temperature and --top_p for generation diversity.
Verify the reward function (GRPO) or reward model path (RLOO) is correct and returns scalar scores.
Windows / PowerShell Specific
Multi-line bash commands with \ will fail in PowerShell. Convert to single-line or use backtick continuation.
Paths with spaces must be quoted in PowerShell.
~ expands correctly in most contexts but verify with $HOME if issues arise.
Verification
Confirm TRL is installed and CLI is available:
trl --help
Expected: prints usage with subcommands sft, dpo, grpo, kto, rloo, reward.
Confirm GPU is visible:
python -c "import torch; print('CUDA:', torch.cuda.is_available(), 'Devices:', torch.cuda.device_count())"
Dry-run a small SFT job (quick smoke test):
trl sft --model_name_or_path Qwen/Qwen2-0.5B --dataset_name trl-lib/Capybara --max_steps 5 --output_dir ./trl-smoke-test
Expected: training starts, logs loss, saves checkpoint to ./trl-smoke-test within a few minutes.
Check output directory after training:
ls ./trl-smoke-test
Expected: config.json, model.safetensors (or adapter files for LoRA), training_args.json, and checkpoint subdirectories.
Load the trained model to confirm it loads correctly:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("./trl-smoke-test" )
tokenizer = AutoTokenizer.from_pretrained("./trl-smoke-test" )
print ("Model loaded successfully:" , model.config.model_type)
Related skills
huggingface-hub — authentication, model upload/download, dataset managementaccelerate — distributed training configuration and launchpeft — LoRA and other parameter-efficient fine-tuning methodstransformers — base model loading, tokenization, inference
Additional Resources
Limitations
Use this skill only when the task clearly matches TRL's CLI scope.
Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.
Never hardcode real API tokens or Hugging Face tokens in scripts or configs — use environment variables or YOUR_KEY placeholders.
