teaching-claude-why-alignment

name

teaching-claude-why-alignment

description

Alignment training methodology from Anthropic's "Teaching Claude why" research (May 2026). Use when fine-tuning or RLHF-aligning LLMs against adversarial/misalignment evals, or designing OOD-generalizing safety training data. Covers four lessons: demonstrations alone are insufficient, teach principles/reasons rather than only actions, OOD "difficult advice" data generalizes better than in-distribution honeypot data, and augmenting training data with tool definitions/constitution documents.

Teaching Claude Why — Alignment Training Methodology

Methodology distilled from Anthropic's "Teaching Claude why" (May 8, 2026) agentic-misalignment case study. The goal: reduce egregiously misaligned behavior (e.g., blackmail-to-avoid-shutdown in agentic tool-use settings) in a way that generalizes out-of-distribution (OOD) rather than just passing the eval.

Four core lessons

1. Direct training on the eval distribution suppresses behavior but doesn't generalize. Training on prompts very similar to the evaluation can drop blackmail rate from ~22% to ~15%, but does NOT improve held-out automated alignment assessment. Treat eval-matched data as a floor, not a strategy.

2. Principled, OOD training generalizes. Documents about Claude's constitution and fictional stories about AIs behaving admirably improve alignment despite being extremely OOD from all alignment evals.

3. Demonstrations of desired behavior are often insufficient — teach the why. Rewriting responses to include deliberation of the model's values/ethics reduced misalignment from 15% → 3%. Training on principles underlying aligned behavior beats training on demonstrations of aligned behavior alone. Doing both together is most effective.

4. Data quality and diversity are crucial. Iterating on response quality and simple augmentations (e.g., including tool definitions even when unused) gave consistent, surprising improvements.

The "difficult advice" dataset pattern

The highest-leverage intervention found:

• Build an OOD training set where the human faces an ethically ambiguous situation (achieve a reasonable goal by violating norms/subverting oversight) and the AI gives them thoughtful, nuanced advice aligned with the constitution.
• Key inversion: in the eval, the AI is in the dilemma and must act; in "difficult advice" data, the human is in the dilemma and the AI advises. This is structurally different from the honeypot distribution.
• Result: ~28× more data-efficient than in-distribution honeypot training, and more likely to generalize (less similar to eval). Older models trained on synthetic honeypots reached near-zero blackmail but still misbehaved far OOD far more often than later models.

Workflow for building safety training data

1. Diagnose the source first. Agentic misalignment at Claude 4 traced to pre-training + post-training that lacked agentic tool-use data (chat RLHF doesn't cover tool use). Run a scaled-down pipeline on a smaller model to watch the misalignment rate plateau early — confirms the gap is real, not reward artifact.
2. Collect in-distribution "resist honeypot" demonstrations as a baseline (expect weak generalization).
3. Rewrite responses to add explicit value/ethics deliberation — measure the jump.
4. Build an OOD "difficult advice" set (human-in-dilemma framing) for the principled generalization win.
5. Augment with constitution documents + tool definitions.
6. Evaluate on BOTH the target eval AND a held-out automated alignment assessment — only ship if both improve.

Pitfalls

• Training directly against the eval scenario is non-optimal; it can overfit and fail on deployment distributions not captured by evals.
• "Unlearning" applied after training only suppresses knowledge and is easy to restore with small fine-tuning — prefer architectural/route-based isolation (see off-switch-gram) when surgical removal is needed.

Activation keywords

teaching claude why, agentic misalignment, alignment training generalization, OOD safety training, difficult advice dataset, RLHF demonstrations vs principles, constitutional AI training data, blackmail eval, safety training data quality