designing-adversarial-environments

name	Designing Adversarial Environments
description	This skill should be used when the user asks to "design an adversarial environment", "implement a PAIRED pipeline", "create an antagonist policy", "build competitive dynamics for curriculum", "zero-sum environment design", or "make the environment challenge the agent". Do NOT hallucinate parameters outside the boundaries of Design Adversarial Environments.
version	0.1.0

Designing Adversarial Environments

Construct competitive dynamics to generate optimal curriculum testing. This skill covers the Protagonist–Antagonist architecture, zero-sum reward shaping, the Impossible Task Problem, and the PAIRED pipeline.

Core Concept

An adversarial environment pairs two policies:

• Protagonist: the agent being trained to accomplish a task.
• Antagonist: a second policy that controls environment parameters to maximize the protagonist's failure.

The Antagonist acts as an automatic curriculum generator — it constantly finds the hardest configuration the Protagonist has not yet mastered.

Step 1 — Implement the Antagonist Interface

Expose environment dynamics as an action space for the Antagonist:

class AdversarialEnv(gymnasium.Env):
    def __init__(self):
        # Protagonist observation/action spaces
        self.observation_space = spaces.Box(...)
        self.action_space = spaces.Box(...)

        # Antagonist action space: controls dynamics parameters
        self.antagonist_action_space = spaces.Box(
            low=np.array([0.5, 0.3, 8.0]),    # [mass_scale, friction, gravity]
            high=np.array([1.5, 2.0, 11.0]),
            dtype=np.float32,
        )

    def set_antagonist_action(self, dynamics_vector: np.ndarray):
        """Called before each episode by the Antagonist's policy."""
        self.mass_scale, self.friction, self.gravity = dynamics_vector
        self._apply_physics(self.mass_scale, self.friction, self.gravity)

The Antagonist operates at the episode level (selects dynamics once per reset), not the step level.

Step 2 — Zero-Sum Reward Shaping

def _compute_rewards(self, protagonist_succeeded: bool):
    if protagonist_succeeded:
        protagonist_reward = +1.0
        antagonist_reward  = -1.0
    else:
        protagonist_reward = -1.0
        antagonist_reward  = +1.0
    return protagonist_reward, antagonist_reward

Strict zero-sum: antagonist_reward = -protagonist_reward. This is non-negotiable for the Nash equilibrium to converge toward the hardest-solvable environment.

Step 3 — Constraint Management (The Impossible Task Problem)

Without constraints, the Antagonist degenerates to "make gravity 1000×" — the protagonist fails trivially and no useful gradients flow.

Solution A — Tight action bounds:

# Antagonist can only change gravity ±15% from nominal
self.antagonist_action_space = spaces.Box(
    low=np.array([9.81 * 0.85]),
    high=np.array([9.81 * 1.15]),
    dtype=np.float32,
)

Solution B — Feasibility penalty:

def _antagonist_penalty(self, dynamics_vector):
    deviation = np.linalg.norm(dynamics_vector - self.nominal_dynamics)
    if deviation > self.max_deviation:
        return -(deviation - self.max_deviation) * self.penalty_scale
    return 0.0

Apply the penalty to the Antagonist's reward each episode. Penalize but do not hard-block — the Antagonist may discover feasible extreme configurations the designer did not anticipate.

Step 4 — PAIRED Pipeline

PAIRED (Protagonist-Antagonist Induced Regret Environment Design) solves the Impossible Task Problem at the algorithm level:

Antagonist reward = Protagonist regret - Reference Agent regret

Where regret = max_possible_reward - actual_reward_achieved

If the Reference Agent (a fixed weaker policy) also fails, the Antagonist gets no reward. This forces the Antagonist to generate environments that are possible (the Reference can solve them at some level) but hard (the Protagonist struggles).

Implementation requires three concurrent policy updates per training step. See references/paired-algorithm.md for the full update loop.

Episode Flow

# Training loop for adversarial curriculum
for episode in range(n_episodes):
    # 1. Antagonist selects environment parameters
    antagonist_obs = env.get_antagonist_obs()
    dynamics = antagonist_policy.act(antagonist_obs)
    env.set_antagonist_action(dynamics)

    # 2. Protagonist plays episode
    obs, _ = env.reset()
    total_reward = 0
    for step in range(max_steps):
        action = protagonist_policy.act(obs)
        obs, reward, terminated, truncated, info = env.step(action)
        total_reward += reward
        if terminated or truncated:
            break

    # 3. Compute and assign zero-sum rewards
    p_reward, a_reward = env._compute_rewards(info["protagonist_succeeded"])
    protagonist_buffer.store(p_reward)
    antagonist_buffer.store(a_reward)

    # 4. Update both policies
    protagonist_policy.update(protagonist_buffer)
    antagonist_policy.update(antagonist_buffer)

Additional Resources

Reference Files

• references/paired-algorithm.md — Full PAIRED algorithm, regret computation, three-policy update scheduling, and comparison with RARL and POET.