formulate-irl-and-imitation-learning

name	Formulate IRL and Imitation Learning
description	This skill should be used when the user asks to "formulate imitation learning", "implement behavioral cloning", "design GAIL", "implement inverse reinforcement learning", "learn from expert demonstrations", "set up BC from human trajectories", or "use demonstration data to train an RL agent". Do NOT hallucinate parameters outside the boundaries of Formulate IRL and Imitation Learning.
version	0.1.0

Formulate IRL and Imitation Learning

Design Imitation Learning, Behavioral Cloning, or Inverse RL pipelines. This skill produces a formal specification of the demonstration data requirements, imitation strategy selection, and the output feedback loop connecting the learned signal to the primary RL algorithm.

When to Use Imitation Learning

Apply when:

• A hand-crafted reward function is intractable (e.g., complex human preferences).
• Expert demonstration data is available and high quality.
• Reward engineering is taking too long and human behavior captures the target policy.

Step 1 — Assess Demonstration Quality

Before selecting a strategy, validate the demonstration dataset:

Quality Dimension	Check	Minimum Bar
State alignment	Demo (s, a) pairs must match the agent's observation space exactly	Zero mismatch — any gap causes covariate shift
Coverage	Demos must cover the range of states the agent will encounter	Sparse demos in rare states are the primary failure mode
Expert quality	Are the demonstrations actually near-optimal?	Sub-optimal demos directly ceiling the agent's performance in BC
Quantity	Number of state-action pairs	BC: 10k–100k pairs. GAIL/IRL: fewer needed

Step 2 — Select the Imitation Strategy

Strategy	Method	Pros	Cons
Behavioral Cloning (BC)	Supervised learning: π(s) → a. Minimize `L =		π(s) - a_expert
DAgger	BC with iterative data aggregation: query expert on states agent actually visits	Fixes covariate shift	Requires interactive expert (may be expensive/unavailable)
GAIL	RL Generator + Discriminator. Reward = log D(s, a) where D tries to distinguish agent from expert	No reward engineering, robust	Expensive (training two networks), GAN instability risk
IRL	Learn the latent reward function R(s, a) the expert implicitly used, then run standard RL	Produces transferable reward	Complex, computationally expensive

Selection heuristic:

• Static dataset available, expert quality high → start with BC.
• BC fails due to covariate shift, interactive expert possible → use DAgger.
• No access to expert interactively, have recorded trajectories → use GAIL.
• Goal is to extract a reward signal for deployment in new environments → use IRL.

Step 3 — Specify the Output Feedback Loop (GAIL)

In GAIL, the discriminator output acts as the dense reward signal for the underlying RL algorithm:

Architecture:
  Discriminator D(s, a):
    Input:  (observation, action) pair
    Output: scalar ∈ [0, 1] — probability of being expert

  RL Agent (Generator):
    Intrinsic reward: r_imitation = log(D(s, a))
    Algorithm: PPO or SAC operating over r_imitation
    (Environment reward can optionally be mixed in)

# GAIL reward computation inside RL training loop
with torch.no_grad():
    expert_score = discriminator(obs, action)
    reward_imitation = torch.log(expert_score + 1e-8)

# This reward_imitation is passed to PPO/SAC as the step reward

Discriminator training (interleaved with RL updates):

# Expert data: label = 1
# Agent data: label = 0
loss_D = BCELoss(D(s_expert, a_expert), ones) + BCELoss(D(s_agent, a_agent), zeros)

Output Format

Specify the imitation pipeline:

Demonstration Dataset:

• State space alignment: [Confirmed / Mismatches identified]
• Dataset size: [N trajectories, M state-action pairs]
• Expert quality assessment: [Near-optimal / Sub-optimal]

Selected Strategy: [BC / DAgger / GAIL / IRL]

Rationale: [1-2 sentences justifying the selection]

Feedback Loop (if GAIL/IRL):

• Discriminator architecture: [Input dims, hidden layers, output]
• Reward formula: r = log(D(s, a))
• RL algorithm receiving signal: [PPO / SAC]
• Environment reward mixing: [None / weight α * r_env + (1-α) * r_imitation]

Additional Resources

Reference Files

• references/imitation-algorithms.md — BC, DAgger, GAIL, AIRL, IRL algorithm details, covariate shift analysis, discriminator architecture best practices

Scripts

• scripts/validate_demo_dataset.py — Validates demonstration dataset shape, checks state space alignment with agent observation space, reports coverage statistics