visplay-self-evolving-vlms

name	visplay-self-evolving-vlms
title	VisPlay: Self-Evolving VLMs from Images via Dual-Role GRPO
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.15661
keywords	["Vision-Language Models","Self-Evolution","GRPO","Automatic Curriculum","Unlabeled Images"]
description	Enable VLMs to self-improve from unlabeled images via dual-role framework—questioner generates challenging visual questions while reasoner answers them, trained jointly with GRPO using difficulty and diversity rewards.

Enable Vision-Language Models to Self-Improve from Unlabeled Images

Vision-language models typically require supervised fine-tuning on annotated data. VisPlay breaks this dependency via a dual-role self-evolution framework: the model splits into an Image-Conditioned Questioner (generates visual questions) and a Multimodal Reasoner (answers them), trained jointly with GRPO. The system autonomously creates an automatic curriculum from unlabeled images, progressively increasing difficulty.

No annotations needed—the questioner and reasoner co-evolve, with the questioner learning to generate harder questions as the reasoner improves, creating a self-reinforcing loop of increasing capability.

Core Concept

Supervised learning for VLMs requires annotated data (image-question-answer triples), which is expensive. VisPlay observes that annotators follow a natural curriculum: they ask easier questions first, then harder ones as models improve. VisPlay automates this:

1. Questioner Role: Given an image, generate a challenging visual question about it
2. Reasoner Role: Answer the question using both image and question context

Both roles start as the same base model. During training:

• Questioner learns to generate questions of optimal difficulty (50% confidence for reasoner)
• Reasoner learns to answer increasingly difficult questions

They're trained jointly with GRPO, using composite rewards (uncertainty reward, diversity, format constraints). This creates an automatic curriculum without human annotation.

Architecture Overview

• Dual-Role Architecture: Single base model splits into Questioner and Reasoner for alternating training
• Questioner Rewards: Uncertainty (target 50% reasoner confidence), diversity (penalize redundancy), format (enforce structure)
• Reasoner Rewards: Binary (correct/incorrect) based on pseudo-labels from majority voting on reasoner outputs
• GRPO Training: Group Relative Policy Optimization for stable reward signal without separate value network
• Curriculum Learning: Automatic difficulty progression as reasoner improves, questioner escalates

Implementation Steps

Step 1: Define Questioner and Reasoner Roles. Split model into two heads.

import torch
import torch.nn as nn

class DualRoleVLM(nn.Module):
    
     ():
        ().__init__()

        
        .backbone = load_vision_language_model(base_model_name)

        
        .questioner_head = nn.Linear(hidden_dim, hidden_dim)
        .reasoner_head = nn.Linear(hidden_dim, hidden_dim)

        
        .decoder = nn.TransformerDecoder(
            decoder_layer=nn.TransformerDecoderLayer(hidden_dim, nhead=),
            num_layers=
        )

        .output_projection = nn.Linear(hidden_dim, )  

     ():
        
        
        img_features = .backbone.encode_image(images)  

        
        questioner_features = .questioner_head(img_features)

        
        questions, logits = ._generate_sequence(
            questioner_features,
            max_length=,
            prompt=
        )

         questions, logits

     ():
        
        
        img_features = .backbone.encode_image(images)
        question_tokens = .backbone.encode_text(questions)

        
        reasoner_features = .reasoner_head(img_features)

        
        combined_features = torch.cat([reasoner_features, question_tokens], dim=)

        
        answers, logits = ._generate_sequence(
            combined_features,
            max_length=,
            prompt=
        )

         answers, logits

     ():
        
        batch_size = features.shape[]
        tokens = []
        logits_list = []

        
         _  (max_length):
            
            output = .decoder(features.unsqueeze())  
            output_logits = .output_projection(output[-])  

            
            next_token = output_logits.argmax(dim=-)
            tokens.append(next_token)
            logits_list.append(output_logits)

            
             (next_token == .get_eos_token()).():
                

         torch.stack(tokens, dim=), torch.stack(logits_list, dim=)