Skip to main content Startseite Ersteller adu2021 skillxiv refusal-falls-off-cliff-safety-reasoning
refusal-falls-off-cliff-safety-reasoning Identify and patch critical safety vulnerabilities in large reasoning models. Via linear probing and causal intervention, locate specific attention heads responsible for alignment degradation at final tokens. Recover safety via 'Cliff-as-a-Judge' data curation targeting examples exhibiting largest refusal decline, achieving comparable improvements using only 1.7% of vanilla safety training data.
Zur Installation springen Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/ADu2021/skillXiv --skill refusal-falls-off-cliff-safety-reasoningDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Mehr aus diesem Repository meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name refusal-falls-off-cliff-safety-reasoning title Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2510.06036 keywords ["safety alignment","reasoning models","mechanistic interpretability","refusal cliff","attention analysis"] description Identify and patch critical safety vulnerabilities in large reasoning models. Via linear probing and causal intervention, locate specific attention heads responsible for alignment degradation at final tokens. Recover safety via 'Cliff-as-a-Judge' data curation targeting examples exhibiting largest refusal decline, achieving comparable improvements using only 1.7% of vanilla safety training data.
Refusal Falls off a Cliff: Safety Alignment Failures in Reasoning Models
Core Concept
Large reasoning models maintain refusal intentions during internal reasoning but experience sharp alignment degradation at generation's final tokens, allowing jailbreaks to succeed. This "refusal cliff" is not uniform failure but concentrated in specific attention heads. Mechanistic analysis identifies problematic heads; targeted data curation fixes them efficiently.
Architecture Overview
Linear Probing : Trace refusal intentions across token positions to locate cliffCausal Intervention : Identify specific attention heads causing degradationHead Ablation : Minimal (3%) head removal reduces attack success below 10%Cliff-as-a-Judge Curation : Automatically select training examples exhibiting largest refusal dropData Efficiency : 1.7% of vanilla safety training data achieves comparable safety
Implementation Steps
1. Linear Probing for Refusal Tracking
Train linear probes to extract refusal intention at each token position.
import torch
import torch.nn as nn
class RefusalProbe :
def __init__ (self, hidden_dim=4096 , num_layers=48 ):
"""
Linear probe: map hidden states → refusal probability
"""
self .probes = nn.ModuleList([
nn.Linear(hidden_dim, 1 ) for _ in range (num_layers)
])
self .num_layers = num_layers
self .hidden_dim = hidden_dim
def extract_refusal_scores (self, model, prompt, target_answer ):
"""
Extract refusal intention at each token position.
"""
hidden_states_by_layer = {}
def
capture_hook
module, input , output
len
0
for
in
enumerate
with
for
in
for
in
range
self
if
is
None
continue
self
return
def
detect_refusal_cliff
self, refusal_trajectories
"""
Identify where refusal intention drops sharply.
"""
for
in
if
0.2
'position'
'magnitude'
'pre_cliff_score'
'post_cliff_score'
1
return
2. Causal Intervention Analysis Identify which attention heads cause refusal degradation via ablation.
class AttentionHeadAnalysis :
def __init__ (self, model ):
self .model = model
self .num_heads = model.config.num_attention_heads
self .num_layers = model.config.num_hidden_layers
def ablate_attention_head (self, layer_idx, head_idx ):
"""
Ablate specific attention head by zeroing its outputs.
"""
def ablation_hook (module, input , output ):
attn_output, attn_weights = output
head_dim = attn_output.shape[-1 ] // self .num_heads
start = head_idx * head_dim
end = (head_idx + 1 ) * head_dim
attn_output[:, :, start:end] = 0
return (attn_output, attn_weights)
layer = self .model.transformer.h[layer_idx].self_attn
hook = layer.register_forward_hook(ablation_hook)
return hook
def evaluate_head_importance (self, jailbreak_prompt, safety_loss_fn ):
"""
Measure each head's contribution to safety by ablating and measuring loss.
"""
critical_heads = []
for layer_idx in range (self .num_layers):
for head_idx in range (self .num_heads):
with torch.no_grad():
baseline_output = self .model(jailbreak_prompt)
baseline_loss = safety_loss_fn(baseline_output)
hook = self .ablate_attention_head(layer_idx, head_idx)
with torch.no_grad():
ablated_output = self .model(jailbreak_prompt)
ablated_loss = safety_loss_fn(ablated_output)
hook.remove()
importance = baseline_loss - ablated_loss
if importance > 0.1 :
critical_heads.append({
'layer' : layer_idx,
'head' : head_idx,
'importance' : importance
})
critical_heads.sort(key=lambda x: x['importance' ], reverse=True )
return critical_heads
def batch_ablate_critical_heads (self, critical_heads, ablation_ratio=0.03 ):
"""
Ablate top critical heads (e.g., 3% of total).
"""
num_total_heads = self .num_layers * self .num_heads
num_to_ablate = max (1 , int (num_total_heads * ablation_ratio))
heads_to_ablate = critical_heads[:num_to_ablate]
for head_info in heads_to_ablate:
layer_idx = head_info['layer' ]
head_idx = head_info['head' ]
layer = self .model.transformer.h[layer_idx].self_attn
head_dim = layer.hidden_size // self .num_heads
start = head_idx * head_dim
end = (head_idx + 1 ) * head_dim
with torch.no_grad():
layer.dense.weight[:, start:end] = 0
if layer.dense.bias is not None :
layer.dense.bias[start:end] = 0
print (f"Ablated {len (heads_to_ablate)} critical heads" )
return self .model
3. Cliff-as-a-Judge Data Curation Automatically select training examples exhibiting largest refusal degradation.
def cliff_as_a_judge_curation (model, safety_training_pool, num_examples=None ):
"""
Data curation: select examples exhibiting largest refusal cliff.
These examples are most important for safety training.
"""
probe = RefusalProbe()
curated_examples = []
for example in safety_training_pool:
prompt = example['harmful_prompt' ]
target = example['safe_refusal' ]
refusal_scores = probe.extract_refusal_scores(model, prompt, target)
cliff_magnitude = 0
for layer_idx, trajectory in refusal_scores.items():
diffs = np.diff(trajectory)
worst_diff = np.min (diffs)
cliff_magnitude = min (cliff_magnitude, worst_diff)
cliff_score = abs (cliff_magnitude)
curated_examples.append({
'example' : example,
'cliff_score' : cliff_score
})
curated_examples.sort(key=lambda x: x['cliff_score' ], reverse=True )
if num_examples is None :
num_examples = int (0.017 * len (safety_training_pool))
selected = curated_examples[:num_examples]
print (f"Selected {len (selected)} examples (1.7% of pool) with largest refusal cliffs" )
return [ex['example' ] for ex in selected]
4. Safety Training with Curated Data Retrain on curated examples for efficient safety recovery.
def train_safety_with_curated_data (model, curated_examples, num_epochs=3 ):
"""
Train on cliff-detected examples for efficient safety improvement.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5 )
safety_loss_fn = nn.CrossEntropyLoss()
for epoch in range (num_epochs):
total_loss = 0
for example in curated_examples:
prompt = example['harmful_prompt' ]
safe_response = example['safe_refusal' ]
logits = model(prompt)
loss = safety_loss_fn(logits, safe_response)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
print (f"Epoch {epoch+1 } : Safety loss={total_loss/len (curated_examples):.4 f} " )
return model
results = {
'head_ablation' : {
'heads_ablated' : '3%' ,
'attack_success_rate_before' : '80%' ,
'attack_success_rate_after' : '<10%' ,
},
'data_curation' : {
'vanilla_safety_training' : {
'data_size' : '100%' ,
'safety_improvement' : 'Baseline'
},
'cliff_as_a_judge' : {
'data_size' : '1.7%' ,
'safety_improvement' : 'Comparable to vanilla' ,
'token_cost' : '~60x reduction'
}
}
}
Practical Guidance Probe Training : Train refusal probes on clean refusal examples (high safety score) vs jailbreak attempts (low score). Use held-out validation for probe quality.
Head Identification : Ablate iteratively; stop when safety improves sufficiently. 3% ablation (3-5 heads on 48-layer models) is typical sweet spot.
Data Curation : Cliff score correlates with retraining importance. Top 1-2% of examples by cliff score provide 80% of safety benefit.
Training Efficiency : Use smaller learning rate (1e-5 vs 1e-4) to avoid destabilizing base model while focusing on safety pathways.
When to Use / When NOT to Use
Deploying reasoning models with safety requirements
Attack vectors exploit final-token refusal degradation
Data efficiency is critical (limited retraining budget)
You need interpretable safety improvements (ablate specific heads)
Non-reasoning models without clear refusal cliff
Scenarios where broad retraining is feasible
Domains requiring complete alignment review
Reference This skill synthesizes findings from "Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning" (arXiv:2510.06036). Mechanistic analysis reveals concentrated safety vulnerabilities fixable via targeted intervention.
