human-like-object-grouping

name	human-like-object-grouping
description	Behavioral benchmark and object-centricity analysis for self-supervised vision transformers. Uses two-dot same/different judgment task with 1020+ trials to measure human object grouping, and proposes Gram matrix alignment as a mechanism for improving behavioral alignment. Use when: vision transformer evaluation, object segmentation, self-supervised learning, DINO models, behavioral neuroscience benchmarks, Gram matrix distillation, human-AI visual alignment, psychophysics.
metadata	{"arxiv_id":"2603.13994","published":"2026-03-14","authors":"Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang, Nikolaus Kriegeskorte, Gregory Zelinsky","tags":["vision-transformer","self-supervised","object-segmentation","behavioral-benchmark","gram-matrix","dino","neuroscience"]}

Human-like Object Grouping in Self-supervised Vision Transformers

arXiv: 2603.13994 (cs.CV, cs.AI, q-bio.NC) Authors: Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang, Nikolaus Kriegeskorte, Gregory Zelinsky (Columbia, HKU, NTU, Stony Brook) Updated: v3, 2026-07-09

Core Insight

Self-supervised vision transformers (especially DINO family) implicitly learn object-centric representations that align with human perceptual grouping behavior. The key mechanism is Gram matrix structure — pairwise feature similarity across image patches encodes object boundaries.

Methodology

Behavioral Benchmark: Two-Dot Paradigm

• Task: Participants judge whether two dots on a natural scene are on the same object or different objects
• Measure: Reaction time (RT) reveals object grouping difficulty
• Scale: 72 participants, 255 images × 4 conditions = 1020 trials (COCO2017 images)
• Conditions: same-close, same-far, different-close, different-far
• Key effect: "Same-object advantage" — faster RTs when dots are on same object

Object-Centricity Metric

• Compute affinity maps: cosine similarity of each patch token with all others
• ROC analysis: True Positive Rate (within-object patches) vs False Positive Rate (outside-object)
• AUC: single number quantifying object-centric structure in representations
• Result: DINOv3 ViT-B achieves highest AUC; deeper layers → stronger object-centricity

Behavioral Prediction

• MLP readout on concatenated patch features from two dot locations
• Predict trial-by-trial RTs, normalized to human noise ceiling
• DINOv3 ViT-B: highest noise-normalized Spearman correlation
• Training objective > architecture in driving object-centricity

Gram Matrix Alignment (Key Mechanism Finding)

• Fine-tune supervised models to match Gram matrix of DINOv3 while maintaining classification
• Result: +8-18pp grouping accuracy, substantial object-centric AUC gains
• Transformers more responsive than ConvNets (self-attention naturally encodes pairwise relations)

Key Results

Model	Architecture	Training	Grouping Acc	Object-Centric AUC
DINOv3 ViT-B	Transformer	Self-supervised	91.9%	Highest
DINOv2 ViT-B	Transformer	Self-supervised	89.0%	High
DINOv3 ConvNeXt-B	Convolutional	Distilled	86.7%	High
MAE ViT-B	Transformer	Self-supervised	80.7%	Medium
IN21k ViT-B	Transformer	Supervised	72.2%	Low
IN1K ConvNeXt-B	Convolutional	Supervised	60.0%	Lowest

• Object-centric AUC ↔ behavioral alignment: Spearman r=0.950, p=0.0001
• Stronger object-centric structure → better human RT prediction across ALL model types

Significance

1. Neuroscience bridge: Self-supervised vision models capture object-level organization similar to human visual cortex
2. Training objective matters more than architecture: DINO > supervised with same architecture
3. Gram matrix as active ingredient: Pairwise feature correlation structure drives object-centricity
4. Distillation path: Supervised models can be improved via Gram alignment without retraining from scratch

Activation Keywords

• vision transformer evaluation, object segmentation, self-supervised learning, DINO, behavioral benchmark
• psychophysics, two-dot paradigm, Gram matrix alignment, object-centric representations
• human-AI visual alignment, patch similarity, affinity maps, behavioral prediction
• perceptual grouping, Gestalt, same-object advantage, noise ceiling