slime-preference-optimization

name	slime-preference-optimization
title	SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.02383
keywords	["Preference Optimization","Reference-Free","Margin-Based","Stability","LLM Alignment"]
description	Optimize model preferences by decoupling preference learning from generation quality. Explicitly maximize chosen response likelihood while using token-level stabilization to prevent quality degradation from over-suppressing rejected responses.

SLIME: Stabilized Preference Optimization

Problem

Margin-based preference optimization methods like DPO and SimPO can degrade the quality of preferred responses while optimizing margins. Over-aggressive suppression of rejected tokens removes valid syntax and reasoning patterns.

A reference-free method that preserves preferred response quality while improving alignment is needed.

Core Concept

SLIME decouples preference learning from generation quality through three components: explicit likelihood anchoring of chosen responses, token-level softplus stabilization preventing probability collapse, and dual-margin optimization with soft and hard constraints.

This ensures the model maintains high probability for preferred continuations while treating rejected responses with measured regularization rather than complete suppression.

Architecture Overview

• Likelihood Anchoring: Maximize log-probability of chosen responses explicitly
• Token-Level Stabilization: Softplus penalty prevents rejected probability collapse
• Hard Margin: Defines victory condition where loss becomes zero
• Soft Margin: Continued gradient signal beyond hard margin
• Reference-Free: No need for reference model like in DPO
• Dual-Margin Design: Balances margin satisfaction with output quality

Implementation

Step 1: Define SLIME Loss Function

Implement anchored likelihood with stabilized margins.

import torch
import torch.nn.functional as F

def slime_loss(chosen_logits, rejected_logits, hard_margin=1.0, soft_margin=0.5):
    """Compute SLIME loss combining likelihood and margin objectives."""
    # Get probabilities
    chosen_probs = F.softmax(chosen_logits, dim=-1)
    rejected_probs = F.softmax(rejected_logits, dim=-1)

    # Component 1: Likelihood anchoring—maximize chosen probability
    # Higher is better, so negative for loss
    chosen_log_prob = torch.log(chosen_probs + 1e-8)
    likelihood_loss = -chosen_log_prob.mean()

    
    margin = chosen_probs - rejected_probs

    
    hard_margin_loss = F.relu(hard_margin - margin).().mean()

    
    soft_margin_penalty = F.softplus(soft_margin - margin).mean()

    
    
    rejected_stability = F.softplus(-torch.log(rejected_probs + )).mean()

    
    total_loss = ( * likelihood_loss +
                   * hard_margin_loss +
                   * soft_margin_penalty +
                   * rejected_stability)

     total_loss