blueprint-system-architecture

name	Blueprint System Architecture
description	This skill should be used when the user asks to "blueprint the system architecture", "map the RL data flow", "design the RL pipeline", "define environment-agent interaction", "architect replay buffer placement", "design rollout and learner layers", or "draw the end-to-end RL system diagram". Do NOT hallucinate parameters outside the boundaries of Blueprint System Architecture.
version	0.1.0

Blueprint System Architecture

Map the end-to-end data flow between Environment, Core RL Loop, and Replay memories. This skill produces a structured architectural diagram and component breakdown for any Reinforcement Learning system, covering process topology, hardware placement, and data flow direction.

Core Components to Define

Every RL system architecture consists of four layers that must be explicitly specified:

Layer	Role	Key Questions
Rollout Worker Layer	Generates experience	Async or sync? SubprocVecEnv or single process?
Experience Buffer Storage Layer	Stores transitions	RAM-local or distributed (e.g. Ape-X)?
Learner / Optimizer Layer	Computes gradients	CPU→GPU batch transfer or GPU-native?
Evaluation / Logging Hook Layer	Tracks metrics	Frequency, checkpointing, W&B/TensorBoard?

Step 1 — Map Component Interactivity

Define the process topology first:

• Single-process: Environment runs in the same process as the agent. Simple, debuggable. Use for prototyping.
• SubprocVecEnv: Multiple environment instances run as separate OS processes. Use when environment computation is the bottleneck.
• Distributed (Ape-X / IMPALA style): Rollout workers run on separate machines and push experience to a central Replay Buffer. Learner pulls batches remotely. Use when scale > 1 machine.

Specify the communication mechanism between components: shared memory, pipes, sockets, or gRPC.

Step 2 — Define Hardware Data Placement

Data placement critically impacts throughput. Apply the following rules:

Standard CPU/GPU split (most systems):

1. Environment step() executes on CPU.
2. Experience (s, a, r, s', done) stored in CPU RAM as NumPy arrays.
3. Sampled mini-batches transferred to GPU via .to(device) only during gradient computation.
4. Policy inference during rollout: run on CPU (single sample, low batch size) or GPU (vectorized envs).

GPU-native physics simulation (Isaac Gym / Brax):

• Environment state tensors live permanently on GPU memory.
• Policy inference and gradient computation both occur on GPU.
• Critical rule: Never call .cpu() or .numpy() on these tensors inside the rollout loop — this triggers a GPU→CPU bus transfer that destroys throughput. Keep tensors on-device until logging.

Step 3 — Specify Replay Buffer Architecture

System Type	Buffer Location	Buffer Class
On-Policy (PPO, A2C)	In-process RAM	RolloutBuffer (cleared after each update)
Off-Policy single machine (SAC, DQN)	In-process RAM	ReplayBuffer (circular, persists across updates)
Distributed Off-Policy (Ape-X)	Centralized Redis / Ray store	PrioritizedReplayBuffer (remote)

Output Format

Generate an ASCII diagram specifying all four layers, then follow with a structured breakdown:

┌─────────────────────────────────────────────┐
│           ROLLOUT WORKER LAYER               │
│  [Env 0] [Env 1] ... [Env N]  (CPU/GPU)     │
│  Generates: (s, a, r, s', done) tuples       │
└────────────────────┬────────────────────────┘
                     │ push experience
┌────────────────────▼────────────────────────┐
│        EXPERIENCE BUFFER STORAGE LAYER       │
│  ReplayBuffer / RolloutBuffer (RAM / Remote) │
│  Capacity: N transitions                     │
└────────────────────┬────────────────────────┘
                     │ sample mini-batch → GPU
┌────────────────────▼────────────────────────┐
│          LEARNER / OPTIMIZER LAYER           │
│  Actor + Critic forward pass (GPU)           │
│  Loss computation + backprop (GPU)           │
│  Parameter sync back to workers              │
└────────────────────┬────────────────────────┘
                     │ metrics
┌────────────────────▼────────────────────────┐
│       EVALUATION / LOGGING HOOK LAYER        │
│  Episode return, entropy, loss, FPS          │
│  Checkpoint saves, W&B / TensorBoard         │
└─────────────────────────────────────────────┘

Additional Resources

Reference Files

• references/distributed-patterns.md — Ape-X, IMPALA, and R2D2 architectural patterns
• references/hardware-placement-rules.md — GPU-native simulation rules (Isaac Gym / Brax)

Scripts

• scripts/validate_architecture.py — Validates that a proposed architecture specification dict contains all required layers and fields