Skip to main content Inicio Creadores vuralserhat86 antigravity-agentic-skills model-finetuning
model-finetuning Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/vuralserhat86/antigravity-agentic-skills --skill model-finetuningEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name model_finetuning router_kit AIKit description Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers. version 1.0.0 author Orchestra Research license MIT tags ["DPO","Fine-Tuning","GRPO","HuggingFace","PPO","Post-Training","Preference Alignment","RLHF","Reinforcement Learning","SFT","TRL","agents","algorithms","artificial intelligence","automation","chatbots","cognitive services","deep learning","embeddings","frameworks","generative ai","inference","large language models","llm","machine learning","model fine-tuning","model finetuning","natural language processing","neural networks","nlp","openai","prompt engineering","rag","retrieval augmented generation","tools","vector databases","workflow automation"] dependencies ["trl","transformers","datasets","peft","accelerate","torch"] metadata {"skillport":{"category":"auto-healed","tags":["model_finetuning","model_finetuning"]}}
TRL - Transformer Reinforcement Learning
Quick start
TRL provides post-training methods for aligning language models with human preferences.
Installation :
pip install trl transformers datasets peft accelerate
Supervised Fine-Tuning (instruction tuning):
from trl import SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B" ,
train_dataset=dataset,
)
trainer.train()
DPO (align with preferences):
from trl import DPOTrainer, DPOConfig
config = DPOConfig(output_dir="model-dpo" , beta=0.1 )
trainer = DPOTrainer(
model=model,
args=config,
train_dataset=preference_dataset,
processing_class=tokenizer
)
trainer.train()
Common workflows
Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
Complete pipeline from base model to human-aligned model.
Copy this checklist:
RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model
Step 1: Supervised fine-tuning
Train base model on instruction-following data:
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B" )
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B" )
dataset = load_dataset("trl-lib/Capybara" , split="train" )
training_args = SFTConfig(
output_dir="Qwen2.5-0.5B-SFT"
4
1
2e-5
10
"epoch"
Step 2: Train reward model
Train model to predict human preferences:
from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig
model = AutoModelForSequenceClassification.from_pretrained(
"Qwen2.5-0.5B-SFT" ,
num_labels=1
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT" )
dataset = load_dataset("trl-lib/ultrafeedback_binarized" , split="train" )
training_args = RewardConfig(
output_dir="Qwen2.5-0.5B-Reward" ,
per_device_train_batch_size=2 ,
num_train_epochs=1 ,
learning_rate=1e-5
)
trainer = RewardTrainer(
model=model,
args=training_args,
processing_class=tokenizer,
train_dataset=dataset
)
trainer.train()
trainer.save_model()
Step 3: PPO reinforcement learning
Optimize policy using reward model:
python -m trl.scripts.ppo \
--model_name_or_path Qwen2.5-0.5B-SFT \
--reward_model_path Qwen2.5-0.5B-Reward \
--dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
--output_dir Qwen2.5-0.5B-PPO \
--learning_rate 3e-6 \
--per_device_train_batch_size 64 \
--total_episodes 10000
from transformers import pipeline
generator = pipeline("text-generation" , model="Qwen2.5-0.5B-PPO" )
prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200 )[0 ]["generated_text" ]
print (output)
Workflow 2: Simple preference alignment with DPO Align model with preferences without reward model.
DPO Training:
- [ ] Step 1: Prepare preference dataset
- [ ] Step 2: Configure DPO
- [ ] Step 3: Train with DPOTrainer
- [ ] Step 4: Evaluate alignment
Step 1: Prepare preference dataset
{
"prompt" : "What is the capital of France?" ,
"chosen" : "The capital of France is Paris." ,
"rejected" : "I don't know."
}
from datasets import load_dataset
dataset = load_dataset("trl-lib/ultrafeedback_binarized" , split="train" )
from trl import DPOConfig
config = DPOConfig(
output_dir="Qwen2.5-0.5B-DPO" ,
per_device_train_batch_size=4 ,
num_train_epochs=1 ,
learning_rate=5e-7 ,
beta=0.1 ,
max_prompt_length=512 ,
max_length=1024 ,
logging_steps=10
)
Step 3: Train with DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct" )
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct" )
trainer = DPOTrainer(
model=model,
args=config,
train_dataset=dataset,
processing_class=tokenizer
)
trainer.train()
trainer.save_model()
trl dpo \
--model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name argilla/Capybara-Preferences \
--output_dir Qwen2.5-0.5B-DPO \
--per_device_train_batch_size 4 \
--learning_rate 5e-7 \
--beta 0.1
Workflow 3: Memory-efficient online RL with GRPO Train with reinforcement learning using minimal memory.
GRPO Training:
- [ ] Step 1: Define reward function
- [ ] Step 2: Configure GRPO
- [ ] Step 3: Train with GRPOTrainer
Step 1: Define reward function
def reward_function (completions, **kwargs ):
"""
Compute rewards for completions.
Args:
completions: List of generated texts
Returns:
List of reward scores (floats)
"""
rewards = []
for completion in completions:
score = len (completion.split())
score += len (set (completion.lower().split()))
rewards.append(score)
return rewards
from transformers import pipeline
reward_model = pipeline("text-classification" , model="reward-model-path" )
def reward_from_model (completions, prompts, **kwargs ):
full_texts = [p + c for p, c in zip (prompts, completions)]
results = reward_model(full_texts)
return [r["score" ] for r in results]
from trl import GRPOConfig
config = GRPOConfig(
output_dir="Qwen2-GRPO" ,
per_device_train_batch_size=4 ,
num_train_epochs=1 ,
learning_rate=1e-5 ,
num_generations=4 ,
max_new_tokens=128
)
Step 3: Train with GRPOTrainer
from datasets import load_dataset
from trl import GRPOTrainer
dataset = load_dataset("trl-lib/tldr" , split="train" )
trainer = GRPOTrainer(
model="Qwen/Qwen2-0.5B-Instruct" ,
reward_funcs=reward_function,
args=config,
train_dataset=dataset
)
trainer.train()
trl grpo \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--dataset_name trl-lib/tldr \
--output_dir Qwen2-GRPO \
--num_generations 4
When to use vs alternatives
Need to align model with human preferences
Have preference data (chosen/rejected pairs)
Want to use reinforcement learning (PPO, GRPO)
Need reward model training
Doing RLHF (full pipeline)
SFT : Have prompt-completion pairs, want basic instruction followingDPO : Have preferences, want simple alignment (no reward model needed)PPO : Have reward model, need maximum control over RLGRPO : Memory-constrained, want online RLReward Model : Building RLHF pipeline, need to score generations Use alternatives instead:
HuggingFace Trainer : Basic fine-tuning without RLAxolotl : YAML-based training configurationLitGPT : Educational, minimal fine-tuningUnsloth : Fast LoRA training
Common issues Issue: OOM during DPO training
Reduce batch size and sequence length:
config = DPOConfig(
per_device_train_batch_size=1 ,
max_length=512 ,
gradient_accumulation_steps=8
)
Or use gradient checkpointing:
model.gradient_checkpointing_enable()
Issue: Poor alignment quality
config = DPOConfig(beta=0.5 )
config = DPOConfig(beta=0.01 )
Issue: Reward model not learning
Check loss type and learning rate:
config = RewardConfig(
learning_rate=1e-5 ,
num_train_epochs=3
)
Ensure preference dataset has clear winners:
Issue: PPO training unstable
config = PPOConfig(
kl_coef=0.1 ,
cliprange=0.1
)
Advanced topics SFT training guide : See references/sft-training.md for dataset formats, chat templates, packing strategies, and multi-GPU training.
Hardware requirements
GPU : NVIDIA (CUDA required)VRAM : Depends on model and method
SFT 7B: 16GB (with LoRA)
DPO 7B: 24GB (stores reference model)
PPO 7B: 40GB (policy + reward model)
GRPO 7B: 24GB (more memory efficient)
Multi-GPU : Supported via accelerateMixed precision : BF16 recommended (A100/H100)
Use LoRA/QLoRA for all methods
Enable gradient checkpointing
Use smaller batch sizes with gradient accumulation
Resources Model Finetuning v1.1 - Enhanced
🔄 Workflow
Aşama 1: Preparation (Data-Centric AI)
Aşama 2: Training (Parameter Efficient)
Aşama 3: Alignment & Evaluation
Kontrol Noktaları Aşama Doğrulama 1 Training Loss düşerken Validation Loss artıyor mu (Overfitting)? 2 Model "Catastrophic Forgetting" yaşıyor mu (Eski yeteneklerini kaybetti mi)? 3 Inference hızı ve belleği (VRAM) deployment ortamına uygun mu?
