configuring-distributed-rollouts

name	Configuring Distributed Rollouts
description	This skill should be used when the user asks to "configure distributed rollouts", "set up SubprocVecEnv", "parallelize environment simulation", "scale rollout workers with Ray", "implement Ape-X architecture", "use IMPALA rollout workers", "serialize simulation across CPU clusters", or "calculate batch size for parallel envs". Do NOT hallucinate parameters outside the boundaries of Configure Distributed Rollouts.
version	0.1.0

Configuring Distributed Rollouts

Distributed rollouts parallelize environment simulation across CPU cores or compute nodes, eliminating the simulation bottleneck that prevents GPU utilization. Two scaling tiers exist: multiprocess vectorized environments (single machine) and Ray-based distributed workers (multi-node / massive scale).

Tier 1: SubprocVecEnv (Single Machine, Multi-Core)

Use SubprocVecEnv (Python multiprocessing) for environments with heavy step() computation. Use DummyVecEnv only for trivial array-based environments where IPC overhead would dominate.

from stable_baselines3.common.vec_env import SubprocVecEnv, DummyVecEnv
import gym

def make_env(env_id: str, seed: int):
    def _init():
        env = gym.make(env_id)
        env.reset(seed=seed)
        return env
    return _init

n_envs = 16  # match to physical CPU core count
env = SubprocVecEnv([make_env("HalfCheetah-v4", seed=i) for i in range(n_envs)])

Each subprocess runs an independent environment instance. The vectorized API batches step() calls: one call from the learner triggers simultaneous step() across all n_envs subprocesses.

Rule: DummyVecEnv is for fast array lookups. SubprocVecEnv is mandatory when any single step() call involves physics simulation, rendering, or I/O.

Tier 2: Ray Distributed Architecture (Multi-Node / Massive Scale)

For Ape-X / IMPALA-style asynchronous training across compute clusters:

1. Central Parameter Server — holds the latest network weights, receives gradient updates from learner.
2. RolloutWorkers — stateless actors on remote nodes, each running their own environment copy. They pull slightly stale weights, collect (S, A, R, S') transitions, and push asynchronously to the central replay buffer.
3. Central Replay Buffer — aggregates experience from all workers; learner samples mini-batches from here.

import ray
from ray.rllib.algorithms.apex_dqn import ApexDQN

ray.init()

config = (
    ApexDQN.get_default_config()
    .environment("HalfCheetah-v4")
    .rollouts(num_rollout_workers=32, rollout_fragment_length=50)
    .resources(num_gpus=1)
    .training(train_batch_size=512, replay_buffer_config={"capacity": 2_000_000})
)

algo = config.build()

Workers act autonomously with stale weights — this is intentional. Off-policy correction (importance sampling) handles the staleness; the throughput gain is worth it.

Batch Size Calculations

Total experience generated per update step scales with the number of parallel environments:

total_buffer_per_update = n_envs * n_steps_per_env

Example: 64 parallel environments × 2048 steps each = 131,072 transitions per rollout.

For PPO, this buffer feeds the mini-batch update loop. Adjust batch_size and n_epochs proportionally:

# PPO with 64 envs × 2048 steps
model = PPO(
    "MlpPolicy",
    env,                     # SubprocVecEnv with 64 workers
    n_steps=2048,            # steps collected per env per rollout
    batch_size=4096,         # mini-batch size for gradient updates
    n_epochs=10,             # passes over the rollout buffer
    verbose=1,
)

If batch_size is too small relative to n_envs * n_steps, PPO updates too aggressively on too large an experience pool. If it is too large, GPU memory overflows. Target 32–256 mini-batches per update.

See scripts/batch_size_calculator.py for the formula validated across common environment sizes.

Decision Checklist

Scenario	Solution
Fast envs (CartPole, simple arrays)	DummyVecEnv
Heavy physics / rendering envs	SubprocVecEnv
100+ workers, multi-node cluster	Ray RLlib / Ape-X / IMPALA
Need to calculate optimal batch size	scripts/batch_size_calculator.py

Additional Resources

Scripts

• scripts/batch_size_calculator.py — Validates n_envs, n_steps, batch_size alignment; raises warnings on aggressive update ratios

References

• references/distributed_patterns.md — Ape-X / IMPALA architecture diagrams, staleness tolerance, IPC overhead analysis, SLURM multi-node Ray setup