| name | Design Hierarchical RL |
| description | This skill should be used when the user asks to "design hierarchical RL", "implement the options framework", "build a meta-controller", "design sub-policies", "handle long-horizon tasks", "decompose tasks into subgoals", or "implement HRL with HIRO or feudal networks". Do NOT hallucinate parameters outside the boundaries of Design Hierarchical RL (HRL) Space. |
| version | 0.1.0 |
Design Hierarchical RL (HRL) Space
Implement options frameworks or meta-controller architectures for long-horizon tasks. This skill produces a formal HRL schema defining the meta-controller execution frequency, sub-goal action space, and worker policy reward function.
When to Use HRL
Apply HRL when both of these conditions hold:
- Dense rewards lead to myopic behavior — the agent optimizes short-term progress but never reaches the distant goal.
- Sparse rewards are unlearnable — the goal is too far from the start for random exploration to ever reach it within the training budget.
If standard single-level RL with shaped rewards is still viable, do not add HRL complexity.
Step 1 — Identify Temporal Abstractions
Break the task into distinct temporal levels. For a robot navigation task:
| Level | Example Sub-Tasks |
|---|
| High-Level (Meta) | "Navigate to room A", "Open door", "Pick up object" |
| Low-Level (Worker) | Joint torque commands to walk, reach, grasp |
Define the temporal ratio: how many low-level steps occur per high-level decision. Typical ratios: 10–100 low-level steps per high-level action.
Step 2 — Design the Meta-Controller (High-Level Policy)
The meta-controller operates at low frequency (every $N$ simulation steps):
- Input (Observation): Global state or compressed representation.
- Output (Action): A discrete or continuous sub-goal $g \in \mathcal{G}$.
- Discrete sub-goals: Select from a fixed set of named tasks ("Walk to Door", "Pick Object A").
- Continuous sub-goals: Output a target state vector in the worker's state space (HIRO approach).
- Reward: Receives environment reward only at the coarse timescale.
- Algorithm: Any standard RL algorithm (PPO, SAC) operating at the slow timescale.
Step 3 — Design Sub-Policies (Low-Level Controllers)
Worker policies operate at high frequency (every 1 simulation step):
- Input (Observation): Local state observation + current sub-goal $g$ from meta-controller.
- Output (Action): Primitive actions (joint torques, pixel movements).
- Reward (Intrinsic): Defined entirely by sub-goal achievement, not environment reward:
intrinsic_reward = -|| current_state - subgoal ||²
The worker is isolated from the sparse environment reward — it only cares about reaching the assigned sub-goal.
Step 4 — Address Non-Stationarity
The meta-controller faces a non-stationary environment because the worker policy changes during training. As the worker improves, the same sub-goal produces different trajectories, invalidating the meta-controller's learned transitions.
HIRO off-policy correction:
Re-label old meta-controller transitions by finding the sub-goal $\hat{g}$ that maximizes the likelihood of the worker's actual observed actions:
$$\hat{g} = \arg\max_{g} \sum_{t} \log \pi_{\text{worker}}(a_t \mid s_t, g)$$
This allows off-policy replay of meta-controller experience even as the worker changes.
Output Format
Define the HRL schema:
Meta-Controller Execution Frequency: Every $[N]$ environment steps
Meta-Controller Action Space:
- Type: [Discrete sub-goal index / Continuous sub-goal vector]
- Dimension: [Number of sub-goals or state dimensions]
- Sub-goals: [List sub-goal names or parameterization]
Worker Policy Reward Function:
- Intrinsic reward formula:
reward = -||s_current - g||² (or equivalent)
- Sub-goal success threshold:
||s_current - g|| < ε
Non-Stationarity Treatment:
- [None / HIRO off-policy correction / Goal relabeling / Fixed sub-goal period]
Additional Resources
Reference Files
references/hrl-algorithms.md — HIRO, Feudal Networks (FuN), Option-Critic, DIAYN — algorithm details and implementation considerations
Scripts
scripts/validate_hrl_schema.py — Validates an HRL schema dict for consistency between meta-controller frequency, sub-goal space, and worker reward
