detecting-reward-hacking

name	Detecting Reward Hacking
description	Aggressively audit W&B metrics and simulation replays to ensure the agent is solving the problem and not exploiting simulation bugs.
version	0.1.0

Detecting Reward Hacking

RL Agents are infamous for mathematically satisfying the objective function by exploiting logical bugs in the environment (e.g., pausing the game permanently to avoid losing points, or moving in circles to farm "progress" rewards). This skill establishes the auditing protocols.

Step 1 — Check Episode Length Anomaly

• If the agent suddenly maximizes episode_reward while the episode_length drops to 1 or terminates instantly, the agent found a way to trigger a false success sequence.
• If episode_length maximizes but reward is abnormally high, the agent might be indefinitely looping a small positive reward action (farming without progressing the actual world goals).

Step 2 — Compare Value Function vs Reality

• Monitor Weights & Biases explicitly for the Critic Network's loss.
• If value_loss spikes or explained_variance goes negative while rewards go up, the agent has found a chaotic, highly unpredictable exploit that the critic cannot mathematically model, but the actor is executing.

Step 3 — Visual Audit (Render Check)

• Pause the automated training loops periodically.
• Load the current best .zip / .pt network weights.
• Render 5 full episodes to screen/video and visually verify the agent's behavior conceptually matches human expectations of solving the actual business problem.

Output Format

If hacking is detected: IMMEDIATELY halt the agent-04-training workflow. Log an alert to return to agent-01-discovery, Step 04 designing-reward-function to mathematically patch the vulnerability using Difference Metrics or new Penalty Terms.

If clear:

Audit Status:
  Reward_Hacking_Detected: [False]
  Visual_Verification: [Passed]