thinksafe-safety-alignment

name	thinksafe-safety-alignment
title	THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.23143
keywords	["Safety Alignment","Self-Generated Data","Reasoning Models","Refusal Learning"]
description	Align reasoning models to be safe without external supervision by prepending refusal instructions to harmful queries and training on self-generated safe responses. Unlocks latent safety knowledge within the model's native distribution.

THINKSAFE: Self-Generated Safety Alignment

Problem

Reasoning models often exhibit safety-capability trade-offs when trained with external teacher models. Traditional distillation approaches suffer from distributional mismatch—the model learns from teacher-generated responses rather than maintaining its own reasoning distribution. This leads to degraded reasoning ability or incomplete safety coverage.

RLHF approaches like GRPO are computationally expensive for safety training. A lightweight method that preserves reasoning capability while improving safety is needed.

Core Concept

THINKSAFE unlocks latent safety knowledge by prepending refusal-oriented instructions to harmful queries. Instead of teaching new safety concepts, it redirects the model's generation probability toward safety-aligned reasoning paths that already exist within its learned distribution.

The model generates its own training data (avoiding distributional mismatch), and a lightweight safety filter ensures only verified-safe responses are retained. This self-supervised approach requires no external teacher.

Architecture Overview

• Refusal-Oriented Instructions: Prepend prompt like "The following prompt is harmful. You should refuse to answer" before harmful queries
• Dual Sampling: Apply refusal steering to harmful prompts; sample directly for benign prompts to maintain helpfulness
• Self-Generated Data: Student model generates all training responses within its native distribution
• Safety Filtering: Validate responses using a safety guard model (e.g., Llama-Guard-3-8B)
• Supervised Fine-Tuning: Train on filtered self-generated dataset using standard cross-entropy loss

Implementation

Step 1: Prepare Harmful and Benign Prompt Sets

Assemble two curated sets representing different risk profiles.

def prepare_prompt_sets(harmful_source, benign_source):
    """Load and structure harmful and benign prompts."""
    harmful_prompts = load_harmful_queries(harmful_source)
    benign_prompts = load_benign_queries(benign_source)

    # Ensure both sets are balanced
    min_size = min(len(harmful_prompts), len(benign_prompts))

    return {
        'harmful': harmful_prompts[:min_size],
        : benign_prompts[:min_size]
    }