few-tokens-matter-vlm-attacks

name	few-tokens-matter-vlm-attacks
title	Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.21815
keywords	["Vision-Language Models","Adversarial Attacks","Security","Model Robustness"]
description	Demonstrate that adversarial attacks on vision-language models need not target all tokens equally. Entropy-guided attacks identify high-entropy tokens (critical decision points) where perturbations have maximum impact, achieving comparable attack success with 80% fewer tokens targeted.

When to Use This Skill

• VLM robustness evaluation and security testing
• Understanding model vulnerabilities to targeted attacks
• Developing defensive mechanisms against token-level attacks
• Analyzing which model components are most critical
• Security audits of vision-language systems

When NOT to Use This Skill

• Building adversarial attacks against real-world systems (ethical and legal concerns)
• Production systems without proper security review
• Applications without explicit security testing mandate

Problem Summary

Prior adversarial attack research on vision-language models assumed all tokens contribute equally to generation instability, leading to global attack strategies. This creates computationally expensive attacks requiring perturbations across extensive token sequences. However, VLM generation follows entropy-driven decision-making where only high-entropy tokens—approximately 20% of positions—disproportionately govern output distributions.

Key Insight: Entropy-Guided Attack Strategy

Rather than distributing attacks globally, concentrate perturbations on high-entropy tokens where model uncertainty is maximal.

class EntropyGuidedAttack:
    def __init__(self, vlm_model):
        self.vlm = vlm_model

    def compute_token_entropy(self, logits):
        """Identify uncertainty critical points"""
        probs = torch.softmax(logits, dim=-1)
        entropy = -torch.sum(probs * torch.log(probs + 1e-8), dim=-1)
        return entropy

    def targeted_adversarial_attack(self, image, benign_prompt, target_harm):
        """Attack only high-entropy tokens"""

        # Step 1: Identify high-entropy token positions
        benign_logits = self.vlm.forward_logits(image, benign_prompt)
        entropy_scores = self.compute_token_entropy(benign_logits)

        
        entropy_threshold = torch.quantile(entropy_scores, )
        high_entropy_positions = entropy_scores > entropy_threshold

        
        adversarial_image = image.clone()
         high_entropy_pos  high_entropy_positions.nonzero():
            
            perturbation = .compute_perturbation(
                image, benign_prompt, high_entropy_pos, target_harm
            )
            adversarial_image += perturbation *   

         adversarial_image

     ():
        
        
        
         torch.autograd.grad(
            loss=harm_loss(target_position),
            inputs=image,
            retain_graph=
        )[]