test-time-rl-vlm

name	test-time-rl-vlm
title	Test-Time Reinforcement Learning for Vision Language Models: TTRV
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.06783
keywords	["test-time-adaptation","reinforcement-learning","vision-language","self-improvement","unlabeled-data"]
description	Adapt vision-language models at inference without labeled data by generating multiple predictions and rewarding high-frequency outputs. Trigger: improve VLM accuracy on deployment with self-generated supervision signals.

Test-Time Reinforcement Learning for Vision Language Models (TTRV)

Core Concept

TTRV enables VLMs to self-improve during inference without any labeled training data. By generating multiple predictions on the same test sample and using frequency-based rewards, the model learns to refine its outputs in real-time. This achieves up to 52.4% accuracy gains on object recognition and matches GPT-4V performance without retraining.

The key insight: When a model produces the same answer multiple times under different sampling conditions, that consensus is a reliable learning signal.

Architecture Overview

• Multiple Sampling: Generate k predictions per test sample with stochastic decoding
• Frequency-Based Rewards: Reward outputs that appear in multiple samples (consensus)
• Entropy Regularization: Penalize overly uncertain distributions to maintain quality
• GRPO Adaptation: Apply Group Relative Policy Optimization at test time
• No Labeled Data Required: Uses model's own outputs as supervision

Implementation Steps

1. Design Frequency-Based Reward Signal

Create a reward function that identifies consensus predictions. Multiple similar outputs suggest higher confidence in that answer.

def frequency_based_reward(predictions, temperature=1.0):
    """
    Compute reward based on prediction frequency across samples.

    Args:
        predictions: List of strings from k independent generations
        temperature: Sharpness of reward (higher = more selective)

    Returns:
        List of rewards, one per prediction
    """
    # Tokenize predictions for similarity matching
    tokens = [set(p.lower().split()) for p in predictions]

    rewards = []
    for i, pred in enumerate(predictions):
        # Count how many other predictions are similar
        # (Jaccard similarity > threshold)
        similarity_threshold = 0.7
        similar_count = 0

        for j, other  (predictions):
             i != j:
                intersection = (tokens[i] & tokens[j])
                union = (tokens[i] | tokens[j])
                jaccard = intersection / (union + )

                 jaccard > similarity_threshold:
                    similar_count += 

        
        base_reward = similar_count / ((predictions) - )

        
        reward = base_reward ** ( / temperature)
        rewards.append(reward)

     torch.tensor(rewards, dtype=torch.float32)