designing-value-critic

name	Designing Value Critic
description	This skill should be used when the user asks to "design a value network", "implement a critic", "build a Q-network", "implement twin critic", "add target networks", "implement Polyak averaging", "build a state-value V(s) network", "build an action-value Q(s,a) network", or "design the critic for SAC/TD3/PPO". Do NOT apply any activation function to the final State-Value output layer.
version	0.1.0

Designing Value Critic

The Value Critic estimates the goodness of states or state-action pairs. Correct architecture prevents overestimation bias, stabilizes Bellman bootstrapping, and avoids the deadly triad of divergence.

State-Value V(s) Critics — On-Policy (PPO)

Receives only the State observation. Output must be a single linear neuron with NO activation function:

import torch.nn as nn

class StateCritic(nn.Module):
    def __init__(self, features_dim: int, hidden_dim: int = 64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(features_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, 1),  # NO activation — unbounded scalar
        )

    def forward(self, features):
        return self.net(features)  # shape: (batch, 1)

No activation on the output. V(s) must be an unbounded float scalar. Applying ReLU would prevent negative value predictions; Tanh would cap predictions at ±1.

Action-Value Q(s,a) Critics — Off-Policy (SAC / TD3)

Receives both State and Action as input. Concatenate the state-latent vector and the action vector at the first hidden layer:

class QCritic(nn.Module):
    def __init__(self, features_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(features_dim + action_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1),  # NO activation — unbounded Q-value scalar
        )

    def forward(self, features, action):
        x = torch.cat([features, action], dim=-1)
        return self.net(x)  # shape: (batch, 1)

Twin-Critic Architecture (Overestimation Bias Correction)

Neural network function approximators inherently overestimate Q-values due to max-operations in the Bellman backup. Implement Two independent Critic Networks, initialized with different weights.

During Bellman updates, take the minimum of both:

class TwinQCritic(nn.Module):
    def __init__(self, features_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        self.q1 = QCritic(features_dim, action_dim, hidden_dim)
        self.q2 = QCritic(features_dim, action_dim, hidden_dim)

    def forward(self, features, action):
        return self.q1(features, action), self.q2(features, action)

    def min_q(self, features, action):
        q1, q2 = self.forward(features, action)
        return torch.min(q1, q2)

Target Q computation:

with torch.no_grad():
    target_q1 = target_critic.q1(next_features, next_action)
    target_q2 = target_critic.q2(next_features, next_action)
    target_q = torch.min(target_q1, target_q2)  # Pessimistic lower bound
    td_target = reward + gamma * (1 - done) * target_q

Use the deterministic script for Twin-Critic construction and Polyak averaging:

• scripts/twin_critic_polyak.py — Full TwinCritic class + Polyak averaging utility

Target Network Freezing + Polyak Averaging

Critics are unstable because they chase a moving target. Maintain a frozen target_critic copy and softly update it every step with Polyak averaging (tau = 0.005):

def polyak_update(source: nn.Module, target: nn.Module, tau: float = 0.005) -> None:
    """
    Soft-update target network parameters.
    target = tau * source + (1 - tau) * target
    """
    with torch.no_grad():
        for src_param, tgt_param in zip(source.parameters(), target.parameters()):
            tgt_param.data.mul_(1.0 - tau)
            tgt_param.data.add_(tau * src_param.data)

Initialize target network as an exact copy and freeze gradients:

import copy

target_critic = copy.deepcopy(critic)
# Freeze target — never call optimizer.step() on target_critic
for param in target_critic.parameters():
    param.requires_grad = False

Bellman Backup Reference

Full TD target computation pattern:

TD_target = r_t + gamma * (1 - done_t) * min(Q1_target(s_{t+1}, a_{t+1}), Q2_target(s_{t+1}, a_{t+1}))
Critic_loss = MSE(Q1(s_t, a_t), TD_target) + MSE(Q2(s_t, a_t), TD_target)

See references/bellman_equations.md for full derivations and algorithm-specific variants.

Architecture Decision Guide

Algorithm	Critic Type	Notes
PPO	V(s) single network	Share or separate from actor
DQN	Q(s, a) — one per discrete action	Output is N-dim vector
SAC	Twin Q(s,a) + target	tau = 0.005
TD3	Twin Q(s,a) + target	tau = 0.005, delayed actor update

Additional Resources

Scripts

• scripts/twin_critic_polyak.py — Twin-Critic class and Polyak averaging utility

References

• references/bellman_equations.md — Bellman equation derivations, TD targets for PPO/SAC/TD3/DQN