Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
Instrucciones de origen · Vista previa de solo lectura
name
verl-rl-training
description
Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
verl: Volcano Engine Reinforcement Learning for LLMs
verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.
When to Use verl
Choose verl when you need:
Production-ready RL training at scale (tested up to 671B parameters)
Use this workflow for training reasoning models on math tasks like GSM8K or MATH.
Prerequisites Checklist
GPU cluster with 8+ GPUs (H100 recommended)
Dataset in parquet format with prompt and reward_model columns
Base model from HuggingFace Hub
Step 1: Prepare Dataset
import pandas as pd
data = [
{
"prompt": [{"role": "user", "content": "What is 15 + 27?"}],
"reward_model": {"ground_truth": "42"}
},
# ... more examples
]
df = pd.DataFrame(data)
df.to_parquet("train.parquet")
Step 2: Define Reward Function
# reward_function.pyimport re
defcompute_reward(responses, ground_truths):
rewards = []
for response, gt inzip(responses, ground_truths):
# Extract answer from responsematch = re.search(r'\\boxed{([^}]+)}', response)
ifmatchandmatch.group(1).strip() == gt.strip():
rewards.append(1.0)
else:
rewards.append(0.0)
return rewards
Step 3: Create Training Config
# config/grpo_math.yamlalgorithm:adv_estimator:grpogamma:1.0lam:1.0data:train_files:/path/to/train.parquetval_files:/path/to/val.parquettrain_batch_size:256max_prompt_length:512max_response_length:2048actor_rollout_ref:model:path:Qwen/Qwen2.5-7B-Instructactor:use_kl_loss:truekl_loss_coef:0.001ppo_mini_batch_size:64rollout:name:vllmn:8# samples per prompttemperature:0.7top_p:0.95trainer:total_epochs:3n_gpus_per_node:8save_freq:100
Use this workflow when you need value-based advantage estimation (GAE).
Key Differences from GRPO
Requires separate critic model
Uses Generalized Advantage Estimation (GAE)
Better for tasks with dense rewards
Configuration
algorithm:adv_estimator:gae# Use GAE instead of GRPOgamma:0.99lam:0.95critic:model:path:Qwen/Qwen2.5-7B-Instruct# Can be same or different from actorppo_mini_batch_size:64actor_rollout_ref:actor:use_kl_loss:truekl_loss_coef:0.02clip_ratio:0.2# PPO clipping
# On head node
ray start --head --port=6379
# On worker nodes
ray start --address='head_ip:6379'# Launch training
python3 -m verl.trainer.main_ppo \
trainer.nnodes=4 \
trainer.n_gpus_per_node=8
Configuration Reference
Algorithm Selection
Algorithm
adv_estimator
Use Case
GRPO
grpo
Critic-free, math/reasoning
PPO/GAE
gae
Dense rewards, value estimation
REINFORCE++
reinforce_plus_plus
Variance reduction
RLOO
rloo
Leave-one-out baseline
ReMax
remax
Maximum reward baseline
OPO
opo
Optimal policy optimization
Key Parameters
# Rollout parametersactor_rollout_ref.rollout.n:8# Samples per promptactor_rollout_ref.rollout.temperature:0.7# Sampling temperatureactor_rollout_ref.rollout.top_p:0.95# Nucleus sampling# Training parametersactor_rollout_ref.actor.lr:1e-6# Learning rateactor_rollout_ref.actor.ppo_mini_batch_size:64actor_rollout_ref.actor.clip_ratio:0.2# PPO clip range# KL controlactor_rollout_ref.actor.use_kl_loss:trueactor_rollout_ref.actor.kl_loss_coef:0.001algorithm.kl_ctrl.target_kl:0.1# For adaptive KL control
Common Issues and Solutions
Issue: OOM During Rollout
Symptoms: CUDA out of memory during generation phase
Solutions:
# Reduce batch sizeactor_rollout_ref.rollout.log_prob_micro_batch_size:4# Enable gradient checkpointingactor_rollout_ref.model.enable_gradient_checkpointing:true# Use FSDP2 with CPU offloadingactor_rollout_ref.actor.strategy:fsdp2actor_rollout_ref.actor.fsdp_config.offload_policy:true
