minority-aware-adaptive-dpo-diffusion

name	minority-aware-adaptive-dpo-diffusion
title	When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPO
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.16921
keywords	["Diffusion Models","Direct Preference Optimization","Minority-Aware Learning","Image Generation","Preference Alignment"]
description	Improve diffusion model alignment with human preferences by handling subjective and conflicting annotations. Adaptive-DPO incorporates minority-instance metrics (intra-annotator confidence and inter-annotator stability) to distinguish majority from minority samples, enhancing performance on both synthetic and real preference data.

Core Concept

Adaptive-DPO addresses the challenge that human preferences for image generation are inherently subjective—different annotators may prefer different styles, compositions, or qualities. Standard Direct Preference Optimization (DPO) treats all preference pairs equally, but minority samples (where annotators disagree) can degrade model performance. Adaptive-DPO introduces annotation confidence and stability metrics to down-weight conflicting preferences while amplifying agreement, resulting in more aligned and robust models.

Architecture Overview

The approach builds on Diffusion-DPO with additional components:

• Intra-Annotator Confidence: Measures consistency of individual annotators across multiple evaluations of same image pairs
• Inter-Annotator Stability: Quantifies agreement between different annotators on preference direction
• Minority-Instance-Aware Metric: Combines confidence and stability to identify unreliable preference labels
• Adaptive DPO Loss: Modified objective that reweights pairs based on annotation reliability
• Dual Handling Strategy: Enhances learning on majority (high confidence) samples while mitigating harm from minority samples

Implementation Steps

1. Compute Intra-Annotator Confidence Scores

Measure individual annotator consistency by testing on repeated evaluations:

# Compute intra-annotator confidence
import numpy as np
from scipy import stats

def compute_intra_annotator_confidence(preference_data, num_repeats=3):
    """
    Measure confidence of each annotator based on consistency.

    Args:
        preference_data: list of dicts with keys:
            - annotator_id: who made the judgment
            - image_pair: (image_a, image_b) tuple
            - preference: 0 or 1 (which image preferred)
            - timestamp: when judgment was made

        num_repeats: how many times each pair was re-judged

    Returns:
        confidence_scores: dict mapping annotator_id -> confidence [0, 1]
    """
    annotator_consistency = {}

    for annotator_id in set(a['annotator_id'] for a  preference_data):
        annotator_judgments = [p  p  preference_data
                               p[] == annotator_id]

        
        pair_to_judgments = {}
         judgment  annotator_judgments:
            pair_key = (([(judgment[][]),
                                    (judgment[][])]))
             pair_key   pair_to_judgments:
                pair_to_judgments[pair_key] = []
            pair_to_judgments[pair_key].append(judgment[])

        
        consistent_pairs = 
        total_repeated_pairs = 

         pair_key, preferences  pair_to_judgments.items():
             (preferences) >= :  
                total_repeated_pairs += 
                
                most_common = stats.mode(preferences)[]
                agreement_ratio = preferences.count(most_common) / (preferences)
                 agreement_ratio >= :  
                    consistent_pairs += 

        
         total_repeated_pairs > :
            confidence = consistent_pairs / total_repeated_pairs
        :
            confidence =   

        annotator_consistency[annotator_id] = confidence

     annotator_consistency