hpsv3-human-preference-score-evaluation

name	hpsv3-human-preference-score-evaluation
title	HPSv3 - Human Preference Score Evaluation for Images
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.03789
keywords	["image-quality-assessment","preference-learning","vision-language-models","human-alignment"]
description	A VLM-based preference scoring system trained on 1.17M annotated comparisons to evaluate text-to-image generation quality at scale. Uses uncertainty-aware ranking loss for fine-grained assessment across diverse images and supports iterative quality improvement through chain-of-human-preference sampling.

HPSv3: Human Preference Score Evaluation

Core Concept

HPSv3 is a comprehensive human preference scoring framework for evaluating text-to-image generation quality. Rather than designing hand-crafted metrics, it learns human preferences directly from large-scale annotated data, enabling robust assessment across diverse image types and quality levels. The approach combines dataset construction, preference modeling via vision-language models, and iterative refinement to provide both an evaluation metric and a method for quality improvement.

Architecture Overview

• HPDv3 Dataset: Curates 1.08M text-image pairs with 1.17M annotated pairwise comparisons spanning state-of-the-art generative models and real-world images
• Preference Model: Vision-language model (VLM) backbone trained with uncertainty-aware ranking loss to predict relative image quality
• Chain-of-Human-Preference (CoHP): Iterative refinement mechanism that selects high-quality images at each generation step without requiring additional data
• Quality Estimation: Produces normalized preference scores reflecting human consensus on image quality

Implementation Steps

Step 1: Prepare Data and Preference Annotations

Construct a diverse dataset of image-text pairs with human preference judgments. The key is capturing comparisons across different quality distributions and model architectures.

# Pseudo-code for preference annotation preparation
import json

def prepare_preference_data(image_pairs, annotations):
    """
    Prepare pairwise preference data from human annotations.

    Args:
        image_pairs: List of (image_a, image_b, text_prompt) tuples
        annotations: List of preference labels (0=tie, 1=a_better, -1=b_better)

    Returns:
        Formatted preference dataset for training
    """
    preference_dataset = []
    for (img_a, img_b, prompt), pref_label in zip(image_pairs, annotations):
        preference_dataset.append({
            "image_a": img_a,
            "image_b": img_b,
            "prompt": prompt,
            "preference": pref_label,
            "confidence": estimate_annotation_confidence(pref_label)
        })
     preference_dataset