design-reward-function

name	Design Reward Function
description	This skill should be used when the user asks to "design the reward function", "define the reward signal", "formulate reward shaping", "prevent reward hacking", "balance sparse and dense rewards", "add step penalties", or "structure multi-objective rewards". Do NOT hallucinate parameters outside the boundaries of Design Reward Function ($R$).
version	0.1.0

Design Reward Function ($R$)

Design dense and sparse reward components that align agent behavior with the true objective while mitigating reward hacking. This skill produces a formal reward pseudo-code structure and a Reward Hacking Risk Assessment.

Step 1 — Define the Primary Sparse Reward

State the ultimate objective as a single terminal reward:

reward_terminal = +100.0  if terminated and success else 0.0
reward_failure  = -100.0  if terminated and failure  else 0.0

Assess sparsity viability:

• If the probability of randomly achieving the goal within max_steps is near zero, the sparse reward alone is unlearnable. Dense shaping is required.
• Rule of thumb: if an untrained random policy cannot reach the goal within ~1% of episodes, the reward is too sparse to train without shaping.

Step 2 — Formulate Dense Reward Shaping

Use potential-based reward shaping to provide dense gradients without altering the optimal policy:

$$r_{\text{shaped}}(s, a, s') = r(s, a, s') + \gamma \cdot \Phi(s') - \Phi(s)$$

where $\Phi(s)$ is the potential function (e.g., negative distance to goal).

Common shaping examples:

# Progress shaping (distance reduction)
reward_progress = (prev_distance_to_target - curr_distance_to_target) * weight_progress

# Heading alignment
reward_alignment = cos(angle_to_goal) * weight_alignment

Critical Rule — Avoid Reward Cycles: Never reward raw proximity without a difference metric. Rewarding reward = -distance allows an agent to step backward and forward infinitely for zero net reward but with policy gradient noise. Always use Δdistance = prev - curr, which yields exactly zero net reward for any closed loop path. Potential-based shaping formally guarantees this property.

Step 3 — Design Step Cost (The Whip)

Add a small constant negative reward per step to incentivize efficiency:

reward_step = -0.01  # Encourages solving faster, discourages stalling

For robotics or physical systems, add actuator cost penalties to preserve hardware:

reward_actuator = -0.001 * sum(action ** 2)  # Penalize large torques

Step 4 — Balance Multi-Objective Reward Magnitudes

All reward components must be scaled to a consistent magnitude. Mixing +5000 terminal rewards with -0.001 step costs causes:

• Q-value / Value function estimates to diverge
• Policy gradient updates to be dominated by a single component

Target episodic return range: Keep total episode return bounded between approximately -100 and +100. Scale individual components accordingly.

Verification check:

# Estimate total reward range
max_episode_reward = reward_terminal + reward_progress_max + 0  # best case
min_episode_reward = reward_failure + 0 + reward_step * max_steps  # worst case
print(f"Return range: [{min_episode_reward}, {max_episode_reward}]")
# Target: magnitude < 1000, ideally < 200

Output Format

Generate the reward function as pseudo-code:

# Reward Function Pseudo-Code
reward_step_penalty    = -0.01
reward_progress        = (prev_distance - curr_distance) * weight_progress
reward_completion      = +100.0 if terminated and success else 0.0
reward_failure         = -100.0 if terminated and failure else 0.0
reward_actuator_cost   = -0.001 * sum(action ** 2)

total_reward = (
    reward_step_penalty
    + reward_progress
    + reward_completion
    + reward_failure
    + reward_actuator_cost
)

Follow immediately with a Reward Hacking Risk Assessment:

Component	Potential Exploit	Mitigation
reward_progress	Agent oscillates back and forth	Use difference metric (Δdistance), not absolute
reward_completion	Agent finds degenerate terminal state	Validate terminal condition is truly goal-aligned
Step penalty	Agent terminates early via self-destruction	Ensure failure penalty > accumulated step penalties

Additional Resources

Reference Files

• references/reward-shaping-theory.md — Potential-based shaping proof, PBRS theorem, inverse RL connections, reward decomposition techniques

Scripts

• scripts/validate_reward_function.py — Simulates reward function with random trajectories to detect potential hacking loops and checks magnitude balance