diagnose-training-failures

name	Diagnose Training Failures
description	This skill should be used when the user asks to "diagnose training failures", "debug policy collapse", "fix gradient vanishing", "address Q-value overestimation", "fix entropy collapse", "solve catastrophic forgetting", "understand why training crashed", "reward is stuck", "agent stops exploring", or "policy performance suddenly dropped". Do NOT hallucinate parameters outside the boundaries of Diagnose Training Failures.
version	0.1.0

Diagnose Training Failures

Identify and prescribe fixes for the three canonical RL training failure modes: Q-value overestimation, policy collapse via entropy starvation, and catastrophic forgetting. Each diagnosis follows a read-the-metrics → root-cause → prescription pattern.

Diagnosis 1 — Q-Value Overestimation

Symptom: value_loss is climbing while eval/mean_reward is flat or declining. Predicted Q-values grow toward infinity while actual per-episode returns stay near baseline.

Root Cause: The Critic bootstraps its own overestimates, creating a positive feedback loop. Common in off-policy algorithms (SAC, TD3, DDPG) without value regularization.

Detection:

# Compare predicted Q to actual Monte Carlo return
mc_return = compute_discounted_return(rewards, gamma=0.99)
q_pred = critic(obs, action)
overestimation_ratio = (q_pred.mean() / mc_return.mean()).item()
# If ratio > 2.0, overestimation is severe

Prescriptions (in order of severity):

1. Enable Twin Critics — take min(Q1, Q2) as the Bellman target
2. Increase Polyak averaging delay: tau = 0.001 → tau = 0.0001
3. Reduce Critic learning rate by 5×
4. Add L2 value regularization: loss_critic += 1e-4 * q_pred.pow(2).mean()

Diagnosis 2 — Policy Collapse / Entropy Starvation

Symptom: policy_entropy drops to 0.0 in the first 50 iterations. The agent commits to its first randomly-explored action and never deviates.

Root Cause: The entropy coefficient $\alpha$ is too low (or zero), allowing the policy to collapse onto a deterministic local optimum before sufficient exploration has occurred.

Detection:

dist = actor.get_distribution(obs)
entropy = dist.entropy().mean().item()
if entropy < 0.1 and global_step < 50_000:
    flag("ENTROPY COLLAPSE — exploration terminated prematurely")

Prescriptions (in order of severity):

1. Increase entropy coefficient: ent_coef = 0.01 → ent_coef = 0.1
2. For SAC: enable automatic entropy tuning (target_entropy = -dim(action_space))
3. Implement Action Masking to block immediately-terminal actions during early training
4. Add an entropy bonus directly to the reward: reward += alpha * entropy

Diagnosis 3 — Catastrophic Forgetting

Symptom: The policy solves the task at 1M steps. At 1.5M steps, performance crashes to zero and never recovers. The training curve forms an inverted-U and stays down.

Root Cause: The Replay Buffer overwrites high-quality early trajectories with more recent (and potentially lower-quality) data. The Critic unlearns correct value estimates for previously mastered states.

Detection:

# Track performance on a fixed held-out eval set of "solved" trajectories
static_eval_reward = evaluate_on_fixed_dataset(policy, solved_trajectories)
if static_eval_reward < 0.5 * peak_static_eval:
    flag("CATASTROPHIC FORGETTING — performance on solved states degraded")

Prescriptions (in order of severity):

1. Implement learning rate decay schedule (cosine or linear annealing to 1e-5)
2. Protect top 5% of trajectories: never overwrite buffer entries with cumulative return above the 95th percentile
3. Reduce learning rate by 10× after peak performance is first reached
4. Increase Polyak averaging delay for the target network

Quick Diagnostic Checklist

Metric	Healthy Range	Failure Signal
value_loss	Decreasing or stable	Unbounded growth
policy_entropy	Slowly decreasing	Instant collapse to 0
q_pred / mc_return	0.9 – 1.5	> 2.0
eval/std_reward	< 30% of mean	> 100% of mean
actor_grad_norm	0.1 – 10.0	> 100 or exactly 0

Additional Resources

Reference Files

• references/failure_modes.md — Extended failure taxonomy, gradient explosion/vanishing, dead neurons, and NaN debugging

Scripts

• scripts/overestimation_check.py — Q-value vs Monte Carlo return comparison with overestimation ratio
• scripts/entropy_monitor.py — Entropy collapse detector with early-stop flag