verl-rl-training

name	verl-rl-training
description	Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
category	ml-training
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Reinforcement Learning","RLHF","GRPO","PPO","Post-Training","Distributed Training"]
dependencies	["verl>=0.3.0","torch>=2.0.0","ray>=2.41.0","vllm>=0.8.2","transformers>=4.40.0"]

verl: Volcano Engine Reinforcement Learning for LLMs

verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.

When to Use verl

Choose verl when you need:

• Production-ready RL training at scale (tested up to 671B parameters)
• Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)
• Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)
• Multi-turn rollout with tool calling for agentic workflows
• Vision-language model RL training

Consider alternatives when:

• You need Megatron-native training → use slime or miles
• You want PyTorch-native abstractions with Monarch → use torchforge
• You only need simple SFT/DPO → use TRL or Axolotl

Key Features

• Training backends: FSDP, FSDP2, Megatron-LM
• Rollout engines: vLLM, SGLang, HuggingFace Transformers
• Algorithms: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO
• Models: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)
• Advanced: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools

Installation

# Option 1: pip install
pip install verl[vllm]  # or verl[sglang] for SGLang backend

# Option 2: Docker (recommended for production)
docker pull verlai/verl:vllm011.latest

# Option 3: From source
git clone https://github.com/volcengine/verl.git
cd verl && pip install -e .[vllm,math]

Quick Start: GRPO Training

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=~/data/gsm8k/train.parquet \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    trainer.n_gpus_per_node=8

Core Architecture

verl uses a HybridFlow programming model separating control flow from computation: