dupo-dual-preference-optimization

name	dupo-dual-preference-optimization
title	DuPO: Dual Preference Optimization for Reliable LLM Self-Verification
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.14460
keywords	["self-verification","dual-learning","preference-optimization","self-supervised-feedback","language-models"]
description	Implement dual preference optimization to generate self-supervised feedback without manual annotation by decomposing tasks into known/unknown components and reconstructing hidden information from model outputs.

DuPO: Dual Preference Optimization for Reliable LLM Self-Verification

Core Concept

DuPO enables large language models to self-verify outputs without manual feedback by leveraging dual learning frameworks. The technique decomposes task inputs into known and unknown components, then constructs complementary reconstruction tasks. The model's ability to reconstruct hidden information from its primary output serves as an intrinsic reward signal. This approach eliminates dependency on expensive labeled data while improving reasoning quality across mathematical, translation, and general reasoning domains.

Architecture Overview

• Input Decomposition: Split task inputs into observable and hidden components
• Dual Task Construction: Create complementary reconstruction tasks (e.g., reverse engineering solutions to recover variables)
• Reconstruction Quality Scoring: Use reconstruction fidelity as self-supervised reward signal
• Preference Optimization: Apply reward signals to optimize model outputs via preference learning
• Non-Invertible Handling: Extend framework to problems without direct mathematical inverses through learned approximations

Implementation Steps

1. Decompose Task Inputs

Create paired representations separating known and unknown components:

def decompose_task(task_input: str, task_type: str) -> tuple[str, str]:
    """
    Decompose task into observable and hidden components.

    For math: full equation becomes observable setup + hidden variables
    For translation: source+target becomes observable pairs + masked segments
    """
    if task_type == "math":
        # Extract equation structure, mask coefficients/variables
        observable = extract_equation_skeleton(task_input)
        hidden = extract_hidden_variables(task_input)
    elif task_type == "translation":
        # Preserve source, mask target segments
        observable = task_input.split("|||")[0]  # source language
        hidden = task_input.split()[]      
     observable, hidden