configuring-replay-buffers

name	Configuring Replay Buffers
description	This skill should be used when the user asks to "configure a replay buffer", "implement PER", "set up prioritized experience replay", "implement HER", "add hindsight experience replay", "build a ring buffer for RL", or "configure sample distributions for off-policy algorithms". Do NOT use for On-Policy algorithms (e.g., PPO/TRPO) as they do not maintain long-term replay buffers.
version	0.1.0

Configuring Replay Buffers

Replay buffers store and sample past (state, action, reward, next_state, done) transitions for off-policy RL training. Correct buffer implementation prevents memory thrashing, ensures efficient sampling, and handles sparse reward environments.

Standard Replay Buffer

Pre-allocate the entire numpy array or torch tensor in RAM at initialization to prevent dynamic resizing memory thrashing:

class ReplayBuffer:
    def __init__(self, capacity, obs_shape, action_dim):
        self.capacity = capacity
        self.ptr = 0
        self.size = 0
        self.obs = np.zeros((capacity, *obs_shape), dtype=np.float32)
        self.next_obs = np.zeros((capacity, *obs_shape), dtype=np.float32)
        self.actions = np.zeros((capacity, action_dim), dtype=np.float32)
        self.rewards = np.zeros((capacity, 1), dtype=np.float32)
        self.dones = np.zeros((capacity, 1), dtype=np.float32)

    def add(self, obs, next_obs, action, reward, done):
        self.obs[self.ptr] = obs
        self.next_obs[self.ptr] = next_obs
        self.actions[self.ptr] = action
        self.rewards[self.ptr] = reward
        self.dones[self.ptr] = done
        self.ptr = (self.ptr + 1) % self.capacity  # ring-buffer wrap
        self.size = min(self.size + 1, self.capacity)

    def sample(self, batch_size):
        idxs = np.random.randint(0, self.size, size=batch_size)
        return (self.obs[idxs], self.next_obs[idxs],
                self.actions[idxs], self.rewards[idxs], self.dones[idxs])

Use a pointer index that wraps modulo the capacity to mimic a ring-buffer. Never use list.append() for large buffers.

Prioritized Experience Replay (PER)

Standard buffers sample uniformly. High-yield transitions (where TD-error is huge) are rarely sampled. PER fixes this by sampling proportionally to transition priority.

Use the SumTree data structure to efficiently sample transitions proportionally to P(i) = p_i^alpha / sum(p_k^alpha) where p_i is the TD-error magnitude.

Because sampling is biased, actively de-bias the network update step by multiplying loss gradients by Importance Sampling (IS) weights.

Use the deterministic script for SumTree and IS weight calculations:

• scripts/per_sumtree.py — SumTree data structure, priority sampling, IS weight computation

Key parameters:

• alpha (0.6): controls how much prioritization is used (0 = uniform)
• beta (0.4 → 1.0): controls IS weight correction; anneal toward 1.0 over training
• epsilon (1e-6): small constant added to TD-errors to prevent zero priority

Hindsight Experience Replay (HER)

Use HER when the task has sparse rewards and goal-conditioned observations. The agent almost never randomly stumbles onto a +1 goal reward.

When an agent misses the target and terminates, artificially modify the sampled batch so the missed final state is defined as the intended goal for that specific trajectory chunk, yielding a synthetic +1 reward and generating valid gradient signals for "how to reach state X".

HER goal substitution strategies:

• future: Replace goal with a randomly selected future state from the same episode (most effective)
• final: Replace goal with the final state of the episode
• episode: Replace goal with a random state from the same episode

See references/her_and_memory_patterns.md for goal-manipulation schemas and implementation patterns.

Decision Checklist

Scenario	Buffer Type
Standard off-policy (SAC, TD3, DQN)	Standard ReplayBuffer
Want to prioritize high-error transitions	PER with SumTree
Sparse rewards + goal-conditioned obs	HER (wrap standard or PER buffer)
On-policy (PPO, TRPO)	No buffer — use rollout storage

Additional Resources

Scripts

• scripts/per_sumtree.py — SumTree implementation, IS weights, PER buffer class

References

• references/her_and_memory_patterns.md — HER goal-manipulation schemas, memory optimization patterns