omni-reward-modeling

name	omni-reward-modeling
title	Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.23451
keywords	["Reward Modeling","Multimodal","Preferences","RLHF","Foundation Model"]
description	Builds generalist reward models evaluating outputs across 5 modalities (text, image, video, audio, 3D) using free-form preference data. Combines discriminative and generative reward modeling approaches. Covers 9 tasks with 317K preference examples, enabling preference-based alignment for diverse output modalities.

Omni-Reward: Multimodal Preference Modeling

Current reward models focus narrowly on text and images, missing alignment opportunities across diverse modalities. Omni-Reward builds a single generalist model that understands preferences for any output modality and free-form preference descriptions beyond binary choices.

The unified architecture combines discriminative and generative modeling to capture nuanced, personalized preferences.

Core Concept

Key innovation: single model learns preferences across modalities and preference formats:

• Discriminative component: learns which output is better
• Generative component: generates preference descriptions
• Multimodal inputs: text, image, video, audio, 3D objects
• Free-form preferences: beyond binary pairs (e.g., "I prefer answers that are concise but detailed")

Architecture Overview

• Shared multimodal encoder (vision + audio + text embeddings)
• Discriminative head: probability of output A > output B
• Generative head: free-form preference description prediction
• Task-specific adaptation layers for domain customization

Implementation Steps

Build a multimodal encoder that can process any input modality. Use separate sub-encoders with shared projection layer:

class MultimodalPreferenceEncoder(nn.Module):
    def __init__(self, hidden_dim=768, num_modalities=5):
        super().__init__()

        # Sub-encoders per modality
        self.text_encoder = TextEncoder(output_dim=hidden_dim)
        self.image_encoder = VisionTransformer(output_dim=hidden_dim)
        self.video_encoder = VideoTransformer(output_dim=hidden_dim)
        self.audio_encoder = AudioTransformer(output_dim=hidden_dim)
        self.object_3d_encoder = Point3DTransformer(output_dim=hidden_dim)

        # Fusion layer
        self.fusion = nn.MultiheadAttention(
            embed_dim=hidden_dim,
            num_heads=12,
            batch_first=
        )

        .norm = nn.LayerNorm(hidden_dim)

     ():
        
        embeddings = []

           modalities:
            embeddings.append(.text_encoder(modalities[]))
           modalities:
            embeddings.append(.image_encoder(modalities[]))
           modalities:
            embeddings.append(.video_encoder(modalities[]))
           modalities:
            embeddings.append(.audio_encoder(modalities[]))
           modalities:
            embeddings.append(.object_3d_encoder(modalities[]))

        
        stacked = torch.stack(embeddings, dim=)
        fused, _ = .fusion(stacked, stacked, stacked)
        output = .norm(fused.mean(dim=))

         output