Menü
Master reward function design - potential-based shaping, hacking patterns, validation
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
Use when assessing codebase architecture and you feel pressure to soften critique, lead with strengths, or frame problems diplomatically - provides evidence-based critical assessment resisting relationship and economic pressures
Use when cataloging technical debt under time pressure and tempted to explain choices instead of delivering document - enforces execution discipline with scoped delivery patterns for partial catalogs
Use when stakeholders pressure you to change technical priorities and you're tempted to compromise on security-first or call it synthesis - enforces risk-based prioritization over stakeholder preferences
Use when you have architecture documentation from system-archaeologist and need critical assessment, refactoring recommendations, or improvement prioritization - routes to appropriate architect specialist skills
Master A2C, A3C, SAC, TD3 - actor-critic methods for continuous control
Master ε-greedy, UCB, curiosity-driven, RND, intrinsic motivation exploration
| name | reward-shaping-engineering |
| description | Master reward function design - potential-based shaping, hacking patterns, validation |
Invoke this skill when you encounter:
This skill provides systematic frameworks and concrete patterns for reward engineering.
Do NOT use for:
Reward design is often the hardest part of RL. The reward function defines the entire objective the agent optimizes. A poorly designed reward either:
The key insight: You're solving an inverse problem. You want an agent that achieves behavior X. You need to specify function R(s,a,s') such that optimal policy under R produces behavior X. This is much harder than it sounds because:
The Problem: You want agent to do X, but reward incentivizes Y.
Example (CartPole):
Example (Robotics):
Pattern: Specify WHAT success looks like, not HOW to achieve it. Let agent find the HOW.
# Anti-pattern: Specify HOW
bad_reward = -0.1 * np.sum(np.abs(action)) # Penalize movement
# Pattern: Specify WHAT
good_reward = (1.0 if grasp_success else 0.0) + (-0.01 * np.sum(action**2))
# Says: Success is good, movements have small cost
# Agent figures out efficient movements to minimize action cost
The Problem: Sparse rewards mean agent can't learn which actions led to success.
Example (Goal Navigation):
Credit Assignment Window:
Short window (< 10 steps): Need dense rewards every 1-2 steps
Medium window (10-100 steps): Reward every 5-10 steps OK
Long window (> 100 steps): Sparse rewards very hard, need shaping
When to Add Shaping:
The Problem: Agent finds unintended loopholes.
Classic Hacking Patterns:
Shortcut Exploitation: Taking unintended path to goal
Side-Effect Exploitation: Achieving side-effect that gives reward
Scale Exploitation: Abusing unbounded reward dimension
Prevention Framework:
def design_robust_reward(s, a, s_next):
# Core task reward
task_reward = compute_task_reward(s_next)
# Anti-hacking penalties
action_penalty = -0.01 * np.sum(a**2) # Penalize unnecessary action
suspension_penalty = check_suspension(s_next) # Penalize weird postures
return task_reward + action_penalty + suspension_penalty
The Problem: Reward magnitude affects gradient flow.
Example:
Task A rewards: 0 to 1000
Task B rewards: 0 to 1
Same optimizer with fixed learning rate:
Task A: Step sizes huge, diverges
Task B: Step sizes tiny, barely learns
Solution: Normalize both to [-1, 1]
Standard Normalization Pipeline:
def normalize_reward(r):
# 1. Clip to reasonable range (prevents scale explosions)
r_clipped = np.clip(r, -1000, 1000)
# 2. Normalize using running statistics
reward_mean = running_mean(r_clipped)
reward_std = running_std(r_clipped)
r_normalized = (r_clipped - reward_mean) / (reward_std + 1e-8)
# 3. Clip again to [-1, 1] for stability
return np.clip(r_normalized, -1.0, 1.0)
You want to:
The Solution: Potential-Based Shaping
The theorem states: If you add shaping reward of form
F(s, a, s') = γ * Φ(s') - Φ(s)
where Φ(s) is ANY function of state, then:
Why This Matters: You can safely add rewards like distance-to-goal without worrying you're changing what the agent should do.
Original MDP has Q-function: Q^π(s,a) = E[R(s,a,s') + γV^π(s')]
With potential-based shaping:
Q'^π(s,a) = Q^π(s,a) + [γΦ(s') - Φ(s)]
= E[R(s,a,s') + γΦ(s') - Φ(s) + γV^π(s')]
= E[R(s,a,s') + γ(Φ(s') + V^π(s')) - Φ(s)]
The key insight: When computing optimal policy, Φ(s) acts like state-value function offset. Different actions get different Φ values, but relative ordering (which action is best) unchanged.
Proof Sketch:
def potential_based_shaping(s, a, s_next, gamma=0.99):
"""
Compute shaping reward that preserves optimal policy.
Args:
s: current state
a: action taken
s_next: next state (result of action)
gamma: discount factor
Returns:
Shaping reward to ADD to environment reward
"""
# Define potential function (e.g., negative distance to goal)
phi = compute_potential(s)
phi_next = compute_potential(s_next)
# Potential-based shaping formula
shaping_reward = gamma * phi_next - phi
return shaping_reward
def compute_potential(s):
"""
Potential function: Usually distance to goal.
Negative of distance works well:
- States farther from goal have lower potential
- Moving closer increases potential (positive shaping reward)
- Reaching goal gives highest potential
"""
if goal_reached(s):
return 0.0 # Peak potential
else:
distance = euclidean_distance(s['position'], s['goal'])
return -distance # Negative distance
Common Mistake:
# WRONG: This changes the optimal policy!
shaping_reward = -0.1 * distance_to_goal
# WHY WRONG: This isn't potential-based. Moving from d=1 to d=0.5 gives:
# Reward = -0.1 * 0.5 - (-0.1 * 1.0) = +0.05
# But moving from d=3 to d=2.5 gives:
# Reward = -0.1 * 2.5 - (-0.1 * 3.0) = +0.05
# Same reward for same distance change regardless of state!
# This distorts value function and can change which action is optimal.
Right Way:
# CORRECT: Potential-based shaping
def shaping(s, a, s_next):
phi_s = -distance(s, goal) # Potential = negative distance
phi_s_next = -distance(s_next, goal)
return gamma * phi_s_next - phi_s
# Moving from d=1 to d=0.5:
# shaping = 0.99 * (-0.5) - (-1.0) = +0.495
# Moving from d=3 to d=2.5:
# shaping = 0.99 * (-2.5) - (-3.0) = +0.475
# Slightly different, depends on state! Preserves policy.
def compute_total_reward(s, a, s_next, env_reward, gamma=0.99):
"""
Combine environment reward with potential-based shaping.
Pattern: R_total = R_env + R_shaping
"""
# 1. Get reward from environment
task_reward = env_reward
# 2. Compute potential-based shaping (safe to add)
potential = -distance_to_goal(s_next)
potential_prev = -distance_to_goal(s)
shaping_reward = gamma * potential - potential_prev
# 3. Combine
total_reward = task_reward + shaping_reward
return total_reward
| Aspect | Sparse Rewards | Dense Rewards |
|---|---|---|
| Credit Assignment | Hard (credit window huge) | Easy (immediate feedback) |
| Learning Speed | Slow (few positive examples) | Fast (constant signal) |
| Reward Hacking | Less likely (fewer targets) | More likely (many targets to exploit) |
| Convergence | Can converge to suboptimal | May not converge if hacking |
| Real-World | Matches reality (goals sparse) | Artificial but helps learning |
Use SPARSE when:
Use DENSE when:
Use HYBRID when:
def reward_function(s, a, s_next, done):
"""
Standard pattern: sparse task reward + potential-based shaping.
This gets the best of both worlds:
- Sparse task reward prevents hacking on main objective
- Dense shaping prevents credit assignment crisis
"""
# 1. Sparse task reward (what we truly care about)
if goal_reached(s_next):
task_reward = 1.0
else:
task_reward = 0.0
# 2. Dense potential-based shaping (helps learning)
gamma = 0.99
phi_s = -np.linalg.norm(s['position'] - s['goal'])
phi_s_next = -np.linalg.norm(s_next['position'] - s_next['goal'])
shaping_reward = gamma * phi_s_next - phi_s
# 3. Combine: Sparse main objective + dense guidance
total = task_reward + 0.1 * shaping_reward
# Scale shaping (0.1) relative to task (1.0) so task dominates
return total
def validate_reward_choice(sparse_reward_fn, dense_reward_fn, env, n_trials=10):
"""
Compare sparse vs dense by checking:
1. Learning speed (how fast does agent improve?)
2. Final performance (does dense cause hacking?)
3. Stability (does one diverge?)
"""
results = {
'sparse': train_agent(sparse_reward_fn, env, n_trials),
'dense': train_agent(dense_reward_fn, env, n_trials)
}
# Check learning curves
print("Sparse learning speed:", results['sparse']['steps_to_50pct'])
print("Dense learning speed:", results['dense']['steps_to_50pct'])
# Check if dense causes hacking
print("Sparse final score:", results['sparse']['final_score'])
print("Dense final score:", results['dense']['final_score'])
# If dense learned faster AND achieved same/higher score: use dense + validation
# If sparse achieved higher: reward hacking detected in dense
Agent finds unintended path to success.
Example (Quadruped):
Detection:
# Test on distribution shift
if test_on_different_terrain(agent) << train_performance:
print("ALERT: Shortcut exploitation detected")
print("Agent doesn't generalize → learned specific trick")
Prevention:
def robust_reward(s, a, s_next):
# Forward progress
progress = s_next['x'] - s['x']
# Requirement: Stay upright (prevents flipping hack)
upright_penalty = -1.0 if not is_upright(s_next) else 0.0
# Requirement: Reasonable movement (prevents wiggling)
movement_penalty = -0.1 * np.sum(a**2)
return progress + upright_penalty + movement_penalty
Agent exploits direct reward signal rather than task.
Example (Oscillation):
Detection:
def detect_oscillation(trajectory):
positions = [s['pole_angle'] for s in trajectory]
# Count zero crossings
crossings = sum(1 for i in range(len(positions)-1)
if positions[i] * positions[i+1] < 0)
if crossings > len(trajectory) / 3:
print("ALERT: Oscillation detected")
Prevention:
def non_hackable_reward(s, a, s_next):
# Task: Balanced pole
balance_penalty = -(s_next['pole_angle']**2) # Reward being centered
# Prevent oscillation
angle_velocity = s_next['pole_angle'] - s['pole_angle']
oscillation_penalty = -0.1 * abs(angle_velocity)
return balance_penalty + oscillation_penalty
Agent maximizes dimension without bound.
Example (Camera Hack):
Detection:
def detect_unbounded_exploitation(training_history):
rewards = training_history['episode_returns']
# Check if rewards growing without bound
if rewards[-100:].mean() >> rewards[100:200].mean():
print("ALERT: Rewards diverging")
print("Possible unbounded exploitation")
Prevention:
# Use reward clipping
def clipped_reward(r):
return np.clip(r, -1.0, 1.0)
# Or normalize
def normalized_reward(r, running_mean, running_std):
r_norm = (r - running_mean) / (running_std + 1e-8)
return np.clip(r_norm, -1.0, 1.0)
def check_for_hacking(agent, train_env, test_envs, holdout_env):
"""
Comprehensive hacking detection.
"""
# 1. Distribution shift test
train_perf = evaluate(agent, train_env)
test_perf = evaluate(agent, test_envs) # Variations of train
if train_perf >> test_perf:
print("HACKING: Agent doesn't generalize to distribution shift")
return "shortcut_exploitation"
# 2. Behavioral inspection
trajectory = run_episode(agent, holdout_env)
if has_suspicious_pattern(trajectory):
print("HACKING: Suspicious behavior detected")
return "pattern_exploitation"
# 3. Reward curve analysis
if rewards_diverging(agent.training_history):
print("HACKING: Unbounded reward exploitation")
return "reward_signal_exploitation"
return "no_obvious_hacking"
Most common shaping pattern. Safe when done with potential-based formula.
def distance_shaping(s, a, s_next, gamma=0.99):
"""
Reward agent for getting closer to goal.
CRITICAL: Use potential-based formula to preserve optimal policy.
"""
goal_position = s['goal']
curr_pos = s['position']
next_pos = s_next['position']
# Potential function: negative distance
phi = -np.linalg.norm(curr_pos - goal_position)
phi_next = -np.linalg.norm(next_pos - goal_position)
# Potential-based shaping (preserves optimal policy)
shaping_reward = gamma * phi_next - phi
return shaping_reward
Help agent learn smooth actions without changing optimal behavior.
def smoothness_shaping(a, a_prev):
"""
Penalize jittery/jerky actions.
Helps with efficiency and generalization.
"""
# Difference between consecutive actions
action_jerk = np.linalg.norm(a - a_prev)
# Penalty (small, doesn't dominate task reward)
smoothness_penalty = -0.01 * action_jerk
return smoothness_penalty
Encourage efficient control.
def efficiency_reward(a):
"""
Penalize excessive control effort.
Makes solutions more robust.
"""
# L2 norm of action (total control magnitude)
effort = np.sum(a**2)
# Small penalty
return -0.001 * effort
Prevent dangerous states (without hard constraints).
def safety_reward(s):
"""
Soft penalty for dangerous states.
Better than hard constraints (more learnable).
"""
danger_score = 0.0
# Example: Prevent collision
min_clearance = np.min(s['collision_distances'])
if min_clearance < 0.1:
danger_score += 10.0 * (0.1 - min_clearance)
# Example: Prevent extreme states
if np.abs(s['position']).max() > 5.0:
danger_score += 1.0
return -danger_score
Add auxiliary reward if:
Don't add if:
You have expert demonstrations but no explicit reward function. How to learn?
Options:
Idea: Expert is optimal under some reward function. Infer what reward structure makes expert optimal.
Expert demonstrations → Infer reward function → Train agent on learned reward
Key insight: If expert is optimal under reward R, then R(expert_actions) >> R(other_actions)
class InverseRLLearner:
"""
Learn reward function from expert demonstrations.
Assumes expert is performing near-optimal policy under true reward.
"""
def __init__(self, state_dim, action_dim):
# Reward function (small neural network)
self.reward_net = nn.Sequential(
nn.Linear(state_dim + action_dim, 64),
nn.ReLU(),
nn.Linear(64, 1)
)
self.optimizer = torch.optim.Adam(self.reward_net.parameters())
def compute_reward(self, s, a):
"""Learned reward function."""
sa = torch.cat([torch.tensor(s), torch.tensor(a)])
return self.reward_net(sa).item()
def train_step(self, expert_trajectories, agent_trajectories):
"""
Update reward to make expert better than agent.
Principle: Maximize expert returns, minimize agent returns under current reward.
"""
# Expert reward sum
expert_returns = sum(
sum(self.compute_reward(s, a) for s, a in traj)
for traj in expert_trajectories
)
# Agent reward sum
agent_returns = sum(
sum(self.compute_reward(s, a) for s, a in traj)
for traj in agent_trajectories
)
# Loss: Want expert >> agent
loss = agent_returns - expert_returns
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
return loss.item()
Use when:
Don't use when:
Reward scale directly affects gradient magnitude and training stability.
# Without normalization
reward_taskA = 1000 * task_metric # Large magnitude
loss = -policy_gradient * reward_taskA # Huge gradients
# With normalization
reward_normalized = reward_taskA / reward_std # Unit magnitude
loss = -policy_gradient * reward_normalized # Reasonable gradients
class RewardNormalizer:
def __init__(self, epsilon=1e-8):
self.mean = 0.0
self.var = 1.0
self.epsilon = epsilon
def update_statistics(self, rewards):
"""Update running mean and variance."""
rewards = np.array(rewards)
# Exponential moving average (online update)
alpha = 0.01
self.mean = (1 - alpha) * self.mean + alpha * rewards.mean()
self.var = (1 - alpha) * self.var + alpha * rewards.var()
def normalize(self, reward):
"""Apply standardization then clipping."""
# 1. Standardize (zero mean, unit variance)
normalized = (reward - self.mean) / np.sqrt(self.var + self.epsilon)
# 2. Clip to [-1, 1] for stability
clipped = np.clip(normalized, -1.0, 1.0)
return clipped
def clip_reward(r, clip_range=(-1.0, 1.0)):
"""
Clip reward to fixed range.
Prevents large reward spikes from destabilizing training.
"""
return np.clip(r, clip_range[0], clip_range[1])
# Usage
def total_reward(task_r, shaping_r):
# Combine rewards
combined = task_r + shaping_r
# Clip combined
clipped = clip_reward(combined)
return clipped
def validate_reward_function(reward_fn, env, agent_class, n_trials=5):
"""
Systematic validation of reward design.
"""
results = {}
# 1. Learning speed test
agent = train_agent(agent_class, env, reward_fn, steps=100000)
success_rate = evaluate(agent, env, n_episodes=100)
results['learning_speed'] = success_rate
if success_rate < 0.3:
print("WARNING: Agent can't learn → reward signal too sparse")
return False
# 2. Generalization test
test_variants = [modify_env(env) for _ in range(5)]
test_rates = [evaluate(agent, test_env, 20) for test_env in test_variants]
if np.mean(test_rates) < 0.7 * success_rate:
print("WARNING: Hacking detected → Agent doesn't generalize")
return False
# 3. Stability test
agents = [train_agent(...) for _ in range(n_trials)]
variance = np.var([evaluate(a, env, 20) for a in agents])
if variance > 0.3:
print("WARNING: Training unstable → Reward scale issue?")
return False
# 4. Behavioral inspection
trajectory = run_episode(agent, env)
if suspicious_behavior(trajectory):
print("WARNING: Agent exhibiting strange behavior")
return False
print("PASSED: Reward function validated")
return True
| Red Flag | Likely Cause | Fix |
|---|---|---|
| Success rate < 10% after 50k steps | Reward too sparse | Add shaping |
| High variance across seeds | Reward scale/noise | Normalize/clip |
| Passes train but fails test | Reward hacking | Add anti-hacking penalties |
| Rewards diverging to infinity | Unbounded reward | Use clipping |
| Agent oscillates/twitches | Per-step reward exploitation | Penalize action change |
| Learning suddenly stops | Reward scale issue | Check normalization |
Rationalization: "I'll add reward for getting closer to goal, it can't hurt" Problem: Without potential-based formula, changes optimal policy Reality Check: Measure policy difference with/without shaping
Rationalization: "Sparse rewards prevent hacking" Problem: Agent can't learn in long-horizon tasks (credit assignment crisis) Reality Check: 10+ steps without reward → need shaping or fail training
Rationalization: "I'll normalize all rewards to [-1, 1]" Problem: Over-normalization loses task structure (goal vs near-goal now equal) Reality Check: Validate that normalized reward still trains well
Rationalization: "I don't know how to specify rewards, I'll learn from demos" Problem: Inverse RL is slow and requires good demonstrations Reality Check: If you can specify reward clearly, just do it
Rationalization: "I'll add smoothness, energy, safety rewards" Problem: Each auxiliary reward is another hacking target Reality Check: Validate each auxiliary independently
Rationalization: "The reward looks right, must be algorithm issue" Problem: Reward design is usually the bottleneck, not algorithm Reality Check: Systematically validate reward using test framework
Rationalization: "Agent succeeded, so reward design was good" Problem: Agent might succeed on hacked solution, not true task Reality Check: Test on distribution shift / different environment variants
Rationalization: "Sparse rewards generalize better" Problem: Sparse rewards just fail to learn in long episodes Reality Check: Compare learning curves and final policy generalization
Rationalization: "If I clip rewards, I lose information" Problem: Unbounded rewards cause training instability Reality Check: Relative ordering preserved after clipping, information retained
Rationalization: "A reward penalty is a reward penalty" Problem: Non-potential-based shaping CAN change optimal policy Reality Check: Prove mathematically that Φ(s') - Φ(s) structure used
def reaching_reward(s, a, s_next, gamma=0.99):
"""
Task: Reach target location.
"""
goal = s['goal']
# Sparse task reward
if np.linalg.norm(s_next['position'] - goal) < 0.1:
task_reward = 1.0
else:
task_reward = 0.0
# Dense potential-based shaping
distance = np.linalg.norm(s_next['position'] - goal)
distance_prev = np.linalg.norm(s['position'] - goal)
phi = -distance
phi_prev = -distance_prev
shaping = gamma * phi - phi_prev
# Efficiency penalty (optional)
efficiency = -0.001 * np.sum(a**2)
return task_reward + 0.1 * shaping + efficiency
def locomotion_reward(s, a, s_next):
"""
Task: Move forward efficiently.
"""
# Forward progress (sparse)
forward_reward = s_next['x_pos'] - s['x_pos']
# Staying alive (don't fall)
alive_bonus = 1.0 if is_alive(s_next) else 0.0
# Energy efficiency
action_penalty = -0.0001 * np.sum(a**2)
return forward_reward + alive_bonus + action_penalty
def multi_objective_reward(s, a, s_next):
"""
Task: Multiple objectives (e.g., reach goal AND minimize energy).
"""
goal_reward = 10.0 * (goal_progress(s, s_next))
energy_reward = -0.01 * np.sum(a**2)
safety_reward = -1.0 * collision_risk(s_next)
# Weight objectives
return 1.0 * goal_reward + 0.1 * energy_reward + 0.5 * safety_reward
Potential-Based Shaping:
F(s,a,s') = γΦ(s') - Φ(s)
Value Function Shift (with shaping):
V'(s) = V(s) + Φ(s)
Optimal Policy Preservation:
argmax_a Q'(s,a) = argmax_a Q(s,a) (same action, different Q-values)
Reward Normalization:
r_norm = (r - μ) / (σ + ε)
Clipping:
r_clipped = clip(r_norm, -1, 1)
The skill addresses these scenarios: