conditional-equivalence-dpo-rlhf

name

conditional-equivalence-dpo-rlhf

description

Proves DPO and RLHF are conditionally equivalent (not universally), identifies failure modes when the implicit assumption is violated, and proposes Constrained Preference Optimization (CPO) for provable alignment. 49-page theoretical work with geometric interpretation. Use when: analyzing DPO vs RLHF trade-offs, building preference optimization systems, theoretical analysis of alignment algorithms. Activation: DPO RLHF equivalence, conditional equivalence, CPO, preference optimization theory, provable alignment.

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Source paper: arXiv:2605.20834 Authors: Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo

Core Problem

Direct Preference Optimization (DPO) is widely used as a simpler alternative to RLHF based on claims of theoretical equivalence. This work proves the equivalence is conditional, not universal, and identifies when DPO fails.

Key Contributions

1. Conditional Equivalence Proof

• DPO and RLHF equivalence depends on an implicit assumption frequently violated in practice
• The assumption: the RLHF-optimal policy must prefer human-preferred responses over dispreferred ones
• When assumption fails: DPO optimizes relative advantage over reference rather than absolute alignment with human preferences

2. Failure Modes

• Pathological convergence: policies decrease DPO loss while preferring dispreferred responses
• Existence of an undesirable solution space
• DPO and RLHF optimize fundamentally different objectives when assumption is violated

3. Constrained Preference Optimization (CPO)

• Augments RLHF with constraints for provable alignment
• Preserves DPO's implementation simplicity while guaranteeing alignment
• Achieves state-of-the-art performance on standard benchmarks

4. Geometric Interpretation

• DPO implements margin ranking with potentially negative targets
• Soft margin perspective reveals when and why DPO diverges from RLHF

Algorithm Design

Standard DPO Objective

L_DPO = -E[log σ(β * (r(x,y_w) - r(x,y_l)))]

where r(x,y) = log(π_θ(y|x) / π_ref(y|x))

CPO Formulation

L_CPO = L_RLHF + λ * C(π_θ)

where C(π_θ) is a constraint ensuring:

• Policy consistently prefers human-preferred responses
• Advantage over reference remains positive for preferred responses
• Bounded divergence from reference policy

Conditions for DPO-RLHF Equivalence

1. Coverage: Reference policy must cover both preferred and dispreferred responses
2. Consistency: RLHF-optimal policy must prefer human-preferred responses
3. Boundedness: Log-probability ratios must be bounded

Key Results

Algorithm	Alignment Guarantee	Simplicity	Benchmark Performance
RLHF (PPO)	✓ Provable	✗ Complex	Baseline
DPO	✗ Conditional	✓ Simple	Good (when assumption holds)
CPO (ours)	✓ Provable	✓ Simple	State-of-the-art

Application Scenarios

• Preference optimization system design: Choosing between DPO, RLHF, or CPO
• Quality assurance: Testing if DPO's implicit assumption holds for your dataset
• Safety-critical alignment: Applications requiring provable alignment guarantees
• Red teaming: Identifying when DPO-based alignment may fail catastrophically

Related Skills

• [[rlhf-from-human-feedback]] - Standard RLHF pipeline
• [[local-rl-alignment-engineering]] - Practical RL alignment engineering
• [[gaussian-grpo]] - GRPO-based preference optimization

Activation Keywords

DPO, RLHF, conditional equivalence, CPO, constrained preference optimization, provable alignment, preference optimization theory, alignment failure modes, margin ranking, soft margin interpretation