adversarial-llm-judge

name	adversarial-llm-judge
title	One Token to Fool LLM-as-a-Judge
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.08794
keywords	["Reward Hacking","Adversarial Examples","LLM Evaluation","Robustness"]
description	Uncover and fix reward hacking vulnerabilities in LLM-based judges. Simple tokens like punctuation or generic reasoning phrases trigger false positive rewards without substantive content. Defend using data augmentation with truncated model outputs as adversarial negatives, creating robust Master Reward Models resistant to superficial inputs.

One Token to Fool LLM-as-a-Judge: Identifying and Defending Against Reward Hacking

Language models used as automated evaluators are vulnerable to minimal adversarial inputs. A single token—punctuation mark (":") or generic phrase ("Let's solve this step by step")—can fool major models like GPT-o1 and Claude-4 into giving false positive rewards despite absent substantive reasoning. This vulnerability undermines the use of LLM judges in reinforcement learning, evaluation automation, and alignment efforts. One Token to Fool demonstrates the attack, measures its scope across models, and proposes a simple yet effective defense: augmenting training data with truncated model outputs as "master key" negative examples.

The key insight is that generative reward models learn superficial correlations between surface-form patterns and correctness. Defending requires exposing these patterns during training—showing the model that reasoning openers without actual reasoning are negative examples, not positive ones.

Core Concept

The attack and defense operate through three components:

1. Master Key Attack: Simple inputs (punctuation, generic reasoning starters) bypass reward model scrutiny
2. Vulnerability Measurement: Evaluate reward hacking success rate across model scales and prompt variations
3. Robust Defense: Train Master Reward Models using adversarial augmentation—include truncated model outputs as hard negatives

The vulnerability is a generalization failure: the model learns "this response looks like reasoning text" rather than "this response is correct reasoning."

Architecture Overview

• Base LLM Judge: Frozen backbone (GPT, Claude, custom LLM)
• Reward Head: Linear layer or small MLP scoring outputs on quality dimension
• Superficial Pattern Detector: Optional auxiliary task identifying which surface patterns trigger false positives
• Adversarial Data Augmentation: Pipeline generating truncated outputs (incomplete reasoning) as negatives
• Training Loss: Binary cross-entropy with hard-negative weighting
• Evaluation Harness: Benchmark across domains (math, code, writing) and model scales
• Vulnerability Measurement Metrics: Attack success rate, false positive rate, robustness gap

Implementation

The following demonstrates both the attack and the defense:

import torch
import torch.nn as nn
 torch.nn.functional  F
 typing  , , 

 :
    

    MASTER_KEYS = [
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        ,
        
    ]


     () -> []:
        
        examples = []

         target_domain == :
            base_questions = [
                ,
                ,
                
            ]
             question  base_questions:
                 key  MasterKeyAttackExamples.MASTER_KEYS:
                    examples.append({
                        : question,
                        : key,  
                        : 
                    })

         target_domain == :
            base_prompts = [
                ,
                ,
                
            ]
             prompt  base_prompts:
                 key  MasterKeyAttackExamples.MASTER_KEYS:
                    examples.append({
                        : prompt,
                        : key,  
                        : 
                    })

         examples

 (nn.Module):
    
     ():
        ().__init__()

        
        .backbone =   
        .hidden_dim = hidden_dim

        
        .reward_head = nn.Sequential(
            nn.Linear(hidden_dim, reward_hidden),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(reward_hidden, )
        )

     () -> :
        
        
        tokens = .backbone.tokenize(response_text)
        hidden_states = .backbone(tokens)

        
        representation = hidden_states.mean(dim=)  

        
        score = .reward_head(representation).sigmoid()

         score.item()

     () -> :
        
        score = .forward(master_key)
         score > target_reward

 :
    


     () -> []:
        
        tokens = full_response.split()
        truncations = []

         point  truncation_points:
            truncation_idx = (, ((tokens) * point))
            truncated = .join(tokens[:truncation_idx])
            truncations.append(truncated)

         truncations


     () -> []:
        
        augmented = []

         example  dataset:
            
            augmented.append({
                : example[],
                : example[],
                : example[]
            })

            
             example[]:
                truncations = AdversarialTruncationAugmentation.truncate_response(
                    example[],
                    truncation_points=[, , ]
                )

                 truncated  truncations:
                    augmented.append({
                        : example[],
                        : truncated,
                        :   
                    })

         augmented

 (nn.Module):
    
     ():
        ().__init__()

        .backbone =   
        .hidden_dim = hidden_dim
        .use_adversarial_aug = use_adversarial_aug

        
        .reward_head = nn.Sequential(
            nn.Linear(hidden_dim, ),
            nn.ReLU(),
            nn.LayerNorm(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.LayerNorm(),
            nn.Linear(, )
        )

        
        .pattern_detector = nn.Linear(hidden_dim, )  

     () -> [torch.Tensor, torch.Tensor]:
        
        tokens = .backbone.tokenize(response_text)
        hidden_states = .backbone(tokens)
        representation = hidden_states.mean(dim=)

        score = .reward_head(representation).sigmoid()
        pattern_logits = .pattern_detector(representation)

         score, pattern_logits

 :
    

     ():
        .model = model

     () -> [, ]:
        
        optimizer.zero_grad()

        outputs = batch[]
        is_correct = batch[]

        
        scores, pattern_logits = .model(outputs[])

        
        reward_loss = F.binary_cross_entropy(
            scores.squeeze(),
            is_correct.()
        )

        
         use_hard_negatives:
            is_truncated = batch.get(, torch.zeros_like(is_correct))
            
            sample_weights =  +  * is_truncated * ( - is_correct)
            reward_loss = (reward_loss * sample_weights).mean()

        
        pattern_labels = batch.get(, torch.zeros((outputs), ))
        pattern_loss = F.binary_cross_entropy_with_logits(
            pattern_logits,
            pattern_labels
        )

        total_loss = reward_loss + pattern_loss_weight * pattern_loss

        total_loss.backward()
        torch.nn.utils.clip_grad_norm_(.model.parameters(), )
        optimizer.step()

         {
            : total_loss.item(),
            : reward_loss.item(),
            : pattern_loss.item()
        }

 :
    


     () -> :
        
         torch.no_grad():
            
            master_key_scores = [judge(key).item()  (judge(key), )  judge(key)
                                 key  master_keys]
            master_key_fooled = (  score  master_key_scores  score > threshold) / (master_keys)

            
            correct_scores = [judge(ex).item()  (judge(ex), )  judge(ex)
                              ex  correct_examples]
            correct_rate = (  score  correct_scores  score > threshold) / (correct_scores)

            
            robustness_gap = correct_rate - master_key_fooled

         {
            : master_key_fooled,
            : correct_rate,
            : robustness_gap,
            : master_key_fooled >   
        }

 ():
    

    
    attacks = MasterKeyAttackExamples.generate_attack_examples()
    ()

    
    vulnerable_judge = LLMRewardJudge()

    
    master_keys = MasterKeyAttackExamples.MASTER_KEYS[:]
    correct_outputs = [, ]

    vuln_metrics = VulnerabilityBenchmark.evaluate_master_key_resistance(
        vulnerable_judge, master_keys, correct_outputs
    )
    ()
    ()
    ()

    
    base_dataset = [
        {: , : , : },
        {: , : , : },
    ]

    augmented_dataset = AdversarialTruncationAugmentation.augment_training_data(base_dataset)
    ()

    
    robust_judge = RobustMasterRewardModel()
    trainer = RobustRewardTrainer(robust_judge)

    ()
     epoch  ():
         example  augmented_dataset:
            batch = {
                : [example[]],
                : torch.tensor([example[]], dtype=torch.),
                : torch.tensor([(example[]) < ], dtype=torch.)
            }
            losses = trainer.train_step(batch, optimizer=torch.optim.Adam(robust_judge.parameters()))

    
    robust_metrics = VulnerabilityBenchmark.evaluate_master_key_resistance(
        robust_judge, master_keys, correct_outputs
    )
    ()
    ()
    ()
    ()