analyze-saliency-and-values

name	Analyze Saliency and Values
description	This skill should be used when the user asks to "analyze saliency", "compute feature saliency", "run Grad-CAM", "apply integrated gradients", "extract attention weights", "visualize what the agent looks at", "inspect temporal attention maps", or "check if the memory architecture is useful". Do NOT hallucinate parameters outside the boundaries of Analyze Saliency and Values.
version	0.1.0

Analyze Saliency and Values

Isolate attention and feature saliency using Grad-CAM or Integrated Gradients to reveal which inputs drive the agent's decisions and whether the memory architecture is earning its compute.

Step 1 — Gradient Saliency Extraction

Push a specific observation forward through the Actor network and backpropagate gradients to the input with respect to the chosen continuous action magnitude:

obs_tensor = torch.tensor(obs, requires_grad=True)
action = actor(obs_tensor)
action[target_dim].backward()
saliency = obs_tensor.grad.abs()

For image-based policies, apply Grad-CAM to the final convolutional feature map:

# See scripts/grad_cam.py for full implementation
gradients = []  # hooked during backward
activations = []  # hooked during forward
cam = (gradients * activations).mean(dim=1).relu()
cam = F.interpolate(cam, size=obs.shape[-2:], mode='bilinear')

Overlay the resulting heatmap on the original frame to highlight which pixels or vector indices drove the decision most.

For continuous vector observations, produce a bar chart of per-dimension gradient magnitudes ranked by importance.

Step 2 — Temporal Attention Weights

If the Algo Engineer implemented a Transformer or LSTM, extract the attention map matrix:

Transformer attention:

# Hook into multi-head attention output_weights
attn_weights = model.transformer.layers[i].self_attn(..., need_weights=True)
# Shape: (batch, heads, seq_len, seq_len)

LSTM hidden state analysis:

# Track hidden state norm over time — large norm = high reliance on that step
hidden_norms = [h.norm().item() for h in hidden_sequence]

Decision rule — evaluate temporal range:

• If the agent relies exclusively on the instant frame (attn_weights[:, :, -1, -1] ≈ 1.0), the memory architecture is useless.
• Inform the Architect: the LSTM/Transformer should be replaced with a simple MLP to save compute and reduce variance.
• Document the finding in the saliency report with the attention matrix visualized as a heatmap.

Step 3 — Integrated Gradients (Baseline Comparison)

Integrated Gradients attribute importance relative to a neutral baseline (e.g., zero observation):

$$\text{IG}_i(x) = (x_i - x'i) \times \int{\alpha=0}^{1} \frac{\partial F(x' + \alpha(x - x'))}{\partial x_i} d\alpha$$

Approximate with 50 interpolation steps. See scripts/integrated_gradients.py for the full loop.

Useful for checking whether the agent responds to physically meaningful features or spurious correlations.

Step 4 — Report

Produce a Saliency Analysis Report containing:

1. Grad-CAM / saliency heatmaps for at least 5 representative states (goal-approach, obstacle-avoidance, failure state)
2. Temporal attention matrix visualization (if memory architecture present)
3. Verdict on memory architecture utility
4. Top-5 most influential observation dimensions with physical interpretation

Additional Resources

Reference Files

• references/saliency_math.md — Grad-CAM and Integrated Gradients derivations, formulas, and edge cases

Scripts

• scripts/grad_cam.py — Full Grad-CAM implementation with forward/backward hooks
• scripts/integrated_gradients.py — Integrated Gradients loop with interpolation