trl-fine-tuning

name	trl-fine-tuning
description	Post-trains LLMs with Hugging Face TRL trainers: SFT, DPO, reward modeling, PPO, GRPO, and RLHF pipelines on CUDA. Use when instruction-tuning, preference-aligning, or running online RL with a reward function. Not for HuggingFace Trainer-only SFT without preferences, Unsloth/Axolotl YAML stacks, or CPU-only training; never skip LoRA/QLoRA on 7B-class models without checking VRAM.
version	1.0.1
author	Orchestra Research
license	MIT
dependencies	["trl","transformers","datasets","peft","accelerate","torch"]
platforms	["linux","macos","windows"]
metadata	{"hermes":{"tags":["Post-Training","TRL","Reinforcement Learning","Fine-Tuning","SFT","DPO","PPO","GRPO","RLHF","Preference Alignment","HuggingFace"]}}

TRL — Transformer Reinforcement Learning Fine-Tuning

Overview

TRL (Transformer Reinforcement Learning) is the HuggingFace library for post-training language models. It provides trainers for supervised fine-tuning (SFT), preference alignment (DPO and variants), reward modeling, and online reinforcement learning (PPO, GRPO, RLOO, OnlineDPO). This skill covers the full lifecycle from base model to human-aligned model, with copy-pasteable commands and progressive disclosure into reference files for advanced topics.

When to Use

Use this skill when any of the following apply:

• You need to instruction-tune a base model with prompt-completion pairs → SFT
• You have preference data (chosen/rejected pairs) and want alignment without a reward model → DPO
• You are building a full RLHF pipeline (SFT → Reward Model → PPO)
• You need to train a reward model to score generations
• You want online RL with a custom reward function, especially under memory constraints → GRPO
• You need PPO, RLOO, or OnlineDPO for maximum control over reinforcement learning

Trigger keywords: fine-tune, fine-tuning, SFT, DPO, PPO, GRPO, RLHF, reward model, preference alignment, post-training, instruction tuning, RL, reinforcement learning, TRL.

Use alternatives instead:

• HuggingFace Trainer — basic fine-tuning without RL or preferences
• Axolotl — YAML-based training configuration workflows
• Unsloth — fast LoRA training for single-GPU scenarios
• LitGPT — educational / minimal fine-tuning

Prerequisites

Installation

pip install trl transformers datasets peft accelerate

For GPU support ensure torch is installed with the correct CUDA build for your platform.

Hardware Requirements

Method	Model Size	Approx. VRAM	Notes
SFT (LoRA)	7B	16 GB	LoRA/QLoRA required
DPO	7B	24 GB	Stores reference model in memory
PPO	7B	40 GB	Policy + reward model + value model