model-finetuning

name	model_finetuning
router_kit	AIKit
description	Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.
version	1.0.0
author	Orchestra Research
license	MIT
tags	["DPO","Fine-Tuning","GRPO","HuggingFace","PPO","Post-Training","Preference Alignment","RLHF","Reinforcement Learning","SFT","TRL","agents","algorithms","artificial intelligence","automation","chatbots","cognitive services","deep learning","embeddings","frameworks","generative ai","inference","large language models","llm","machine learning","model fine-tuning","model finetuning","natural language processing","neural networks","nlp","openai","prompt engineering","rag","retrieval augmented generation","tools","vector databases","workflow automation"]
dependencies	["trl","transformers","datasets","peft","accelerate","torch"]
metadata	{"skillport":{"category":"auto-healed","tags":["model_finetuning","model_finetuning"]}}

TRL - Transformer Reinforcement Learning

Quick start

TRL provides post-training methods for aligning language models with human preferences.

Installation:

pip install trl transformers datasets peft accelerate

Supervised Fine-Tuning (instruction tuning):

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()

DPO (align with preferences):

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()

Common workflows

Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir=,
    per_device_train_batch_size=,
    num_train_epochs=,
    learning_rate=,
    logging_steps=,
    save_strategy=
)


trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer
)
trainer.train()
trainer.save_model()