unbiased-recovery-policy-gradient

name

unbiased-recovery-policy-gradient

description

SafeExplorer - drop-in PPO modification for RL with a deterministic recovery policy that prevents silent bias in on-policy updates. Uses score-function estimator only at safe timesteps, never evaluates recovery-policy density, so it stays valid where importance sampling breaks. Use when training RL agents on real robots / safety-critical systems where falls/crashes are costly and a separate recovery controller is engaged inside the safe region.

Unbiased Policy Gradient with Recovery Interventions (SafeExplorer, arXiv:2607.08925)

A drop-in fix for PPO when a separate recovery policy takes over whenever the agent leaves a designer-specified safe region. Standard mixed-policy rollouts silently bias every on-policy update; importance-sampling correction is ill-defined when the recovery policy is deterministic.

When to use

• RL on physical robots / safety-critical sims where a fall/crash is very costly.
• You already (or plan to) hand control to a recovery policy inside a safe region.
• You want to minimize training-time failures, NOT trade them off against return (unlike CMDP).

The problem

• Mixed rollout: agent acts, recovery policy intervenes near the safe-region boundary.
• On-policy estimators (PPO) assume a single policy generated the data → biased gradients.
• Importance sampling (IS) correction needs π_recovery(a|s) — undefined when recovery is deterministic, and unstable even when stochastic.

The fix (core estimator)

• Unbiased policy-gradient estimator: use the score function ∇log π_agent(a|s) ONLY at safe timesteps (where the agent, not recovery, acted). Never evaluate or divide by the recovery policy's density.
• This is valid even when recovery is deterministic — exactly where IS breaks — and empirically dominates IS when recovery is stochastic too.

Two acceleration components (because recovery slows credit assignment near the boundary)

1. Closed-form value for recovery-triggering states when dynamics + recovery are deterministic: plug the known recovery outcome into the value target instead of bootstrapping through unknown transitions.
2. Imitation loss: copy recovery actions ONLY when recovery succeeds (not on failed recoveries, which would teach the agent bad behavior).

Implementation steps

1. Define safe region S ⊂ state space (a subset the agent should stay within).
2. Keep a separate recovery policy π_rec (can be a simple controller / scripted).
3. During rollout: agent acts in S; when s ∉ S, π_rec takes over until back in S.
4. PPO modification: mask the policy-gradient term to safe timesteps only: g = E[ 1[s_t ∈ S] · ∇log π_agent(a_t|s_t) · A_t ] (advantage A_t from the value net).
5. Add closed-form value target for boundary states + success-gated imitation loss.
6. Train as usual; tune the safe-region size (too small → frequent recovery → slow; too large → unsafe).

Results (paper, 3 envs × 5 seeds)

• Training-time falls reduced 233× / 48× / 26× on HalfCheetah / Ant / Unitree Go1 vs PPO.
• Matches or exceeds PPO final reward.
• On Ant (unreliable recovery) it is the ONLY method reaching 80% of best final reward.

Pitfalls

• Safe region must be cheap to evaluate each step; complex S adds overhead.
• If recovery frequently fails, the imitation loss gate is essential — without it you teach failure.
• The estimator assumes recovery transitions don't appear in the agent's gradient; if recovery is stochastic AND you want its gradients, this method explicitly drops them (by design).
• Doesn't replace a real safety filter; it's a training-time bias fix, not a runtime guarantee.

Verification

• Compare training-time failure count, final reward, and sample efficiency vs PPO + naive IS.
• Confirm advantage estimates are unbiased on a toy MDP with known optimum.