design-deterministic-eval-loop

name	Design Deterministic Eval Loop
description	This skill should be used when the user asks to "design an eval loop", "build a deterministic evaluation loop", "disable stochastic sampling during eval", "run deterministic policy inference", "evaluate agent performance properly", "set up isolated evaluation environments", "compute mean and std of episode rewards", or "run N-episode policy assessment". Do NOT hallucinate parameters outside the boundaries of Design Deterministic Eval Loop.
version	0.1.0

Design Deterministic Eval Loop

Build isolated, reproducible evaluation loops with stochastic sampling disabled to produce statistically meaningful performance estimates. Training and evaluation environments must never share state.

Step 1 — Deterministic Action Query

Expose a predict(obs, deterministic=True) method on the Actor network. Never call dist.sample() during evaluation:

Continuous policies (Gaussian):

def predict(self, obs, deterministic=False):
    mean, log_std = self.forward(obs)
    if deterministic:
        return mean  # output exactly μ — no noise
    dist = Normal(mean, log_std.exp())
    return dist.rsample()

Discrete policies (Categorical):

def predict(self, obs, deterministic=False):
    logits = self.forward(obs)
    if deterministic:
        return logits.argmax(dim=-1)  # greedy
    dist = Categorical(logits=logits)
    return dist.sample()

Step 2 — Isolated Evaluation Environment

Instantiate a completely separate Gym environment for evaluation — never reuse the training environment:

# Training env — never touch during eval
train_env = make_vec_env("HalfCheetah-v4", n_envs=8, seed=42)

# Eval env — separate instance, separate seed, never reset during training
eval_env = gym.make("HalfCheetah-v4")
eval_env = gymnasium.wrappers.RecordEpisodeStatistics(eval_env)

Why isolation is mandatory: Calling reset() or stepping the training environment during evaluation corrupts the buffer trajectory indices and GAE advantage estimates mathematically. Cross-contamination produces misleading training curves.

Step 3 — Multi-Episode Assessment

Single-episode performance is meaningless in RL due to environment stochasticity. Run a minimum of $N$ independent episodes:

def evaluate_policy(policy, env, n_episodes=100, deterministic=True):
    episode_rewards = []
    for ep in range(n_episodes):
        obs, _ = env.reset(seed=ep)  # deterministic seed per episode
        done = False
        total_reward = 0.0
        while not done:
            with torch.no_grad():
                action = policy.predict(obs, deterministic=deterministic)
            obs, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated
            total_reward += reward
        episode_rewards.append(total_reward)
    
    mean_r = np.mean(episode_rewards)
    std_r  = np.std(episode_rewards)
    return mean_r, std_r, episode_rewards

Minimum episode counts:

• Fast development: $N=10$ (high variance, directional signal only)
• Standard benchmarking: $N=100$ (publishable estimates)
• Deployment qualification: $N=500$ (safety-critical systems)

Interpretation of std:

• Low std (< 10% of mean): stable, reliable policy
• High std (> 50% of mean): brittle policy — performance is luck-dependent, not skill-dependent

Step 4 — Logging and Checkpointing

# Log to W&B or local CSV after every eval cycle
eval_metrics = {
    "eval/mean_reward": mean_r,
    "eval/std_reward": std_r,
    "eval/min_reward": min(episode_rewards),
    "eval/max_reward": max(episode_rewards),
    "eval/n_episodes": n_episodes,
    "global_step": global_step,
}

Save checkpoints only based on eval/mean_reward, not training loss. Training loss decreasing while eval reward stagnates is the primary sign of overfitting to replay buffer distribution.

Additional Resources

Reference Files

• references/eval_loop_patterns.md — Vectorized eval, async eval pipelines, and seeding strategies

Scripts

• scripts/eval_loop.py — Complete deterministic eval loop with logging, seeding, and stats reporting