| name | multibind-attribute-misbinding-benchmark |
| title | MultiBind: Attribute Misbinding in Multi-Subject Generation |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.21937 |
| keywords | ["Multi-Subject Generation","Attribute Binding","Evaluation Protocol","Confusion Matrix","Diagnostic Patterns"] |
| description | Evaluate multi-reference image generation fidelity using MultiBind's dimension-wise confusion framework. Detects cross-subject attribute errors that holistic metrics (FID, CLIP) miss, including drift (degradation), swap (permutation), dominance (interference), and blending (averaging). Protocol uses specialist models for face identity, appearance, pose, and expression; achieves reproducible failure diagnosis revealing severe binding failures in models appearing competitive on aggregate quality. |
| category | Evaluation Infrastructure |
Problem Statement
Existing evaluation of multi-reference generation systems relies on global image quality metrics (FID, CLIP similarity) that conflate controllability with aesthetic quality. These metrics fail to answer diagnostic questions: "Which subject confused with which?" or "Did the model swap attributes or degrade quality?" This obscures fine-grained generation failures in multi-person scene synthesis, where visual attribute coherence is critical.
Dataset Construction
MultiBind comprises 508 instances drawn from real photographs with 1,527 human subjects. Each instance is annotated with:
- Per-subject ground-truth masks and bounding boxes: Precise segmentation isolating each subject region for controlled comparison
- Subject references via canonical transformation: Generatively normalize subject appearance to reduce inherent visual similarity confounds when computing confusion baselines
- Inpainted background references: Provide background context without subject interference
- Entity-indexed prompts: Average 474 words per instance, encoding detailed descriptions for each subject to enable fine-grained control
This construction enables isolating generation errors from inherent subject similarity biases.
Evaluation Protocol: Dimension-Wise Confusion Framework
The protocol decomposes multi-subject generation fidelity across independent dimensions, each evaluated by specialist models:
Face Identity Evaluation: InsightFace embeddings compute per-subject identity consistency and cross-subject confusion matrices, measuring whether face IDs remain bound to intended subjects.
Appearance Evaluation: Qwen3-VL embeddings capture clothing, texture, and visual style, detecting whether appearance attributes transfer between subjects.
Pose Evaluation: ViTPose extracts skeleton keypoints measuring body configuration consistency, revealing whether poses stay with intended subjects or drift.
Expression Evaluation: Qwen3-VL expression embeddings measure facial expression binding, detecting emotional attribute leakage across subjects.
Baseline-corrected similarity matrices isolate generation-induced changes from inherent subject similarity, enabling fair comparison across different subject pairs.
Diagnostic Failure Patterns
The framework identifies four distinct failure modes:
Drift: Subject visual quality degrades without confusing with other subjects. Indicates generation instability on that reference rather than cross-subject interference.
Swap: Attributes permute between subjects in systematic ways. Suggests the model assigns correct attributes but to wrong subject positions.
