Detect and eliminate data contamination that invalidates RL benchmarks by measuring benchmark reconstruction ability. Implement clean evaluation datasets to distinguish genuine reasoning improvements from memorization. Use when validating RL training results or ensuring benchmark integrity for mathematical reasoning.
Detect and eliminate data contamination that invalidates RL benchmarks by measuring benchmark reconstruction ability. Implement clean evaluation datasets to distinguish genuine reasoning improvements from memorization. Use when validating RL training results or ensuring benchmark integrity for mathematical reasoning.
Detecting Contamination in RL Benchmarks: Distinguishing Reasoning from Memorization
RL training on mathematical reasoning benchmarks can show spurious improvements with incorrect reward signals, not from genuine reasoning capability but from memorization of benchmark problems in pre-training data. The Qwen2.5 model family exhibited unexpected performance gains even with random or inverted rewards on standard benchmarks (MATH-500, AIME), while Llama models did not. Investigation reveals that Qwen's pre-training corpus contains these benchmark problems, enabling models to memorize rather than reason.
This contamination invalidates benchmark-based evaluation: a model can appear to improve through RL when it's actually learning to retrieve memorized solutions more reliably. The solution is detecting contamination via activation-based metrics and building clean, procedurally-generated benchmarks that guarantee no data overlap.
Core Concept
The method detects contamination through two key metrics: partial-prompt completion rate (how well models reconstruct full problems from 60% of the prompt using ROUGE-L) and partial-prompt answer accuracy (whether models recover correct answers). Models with high reconstruction ability are memorizing, not reasoning.
The RandomCalculation dataset provides a clean alternative: procedurally-generated arithmetic expressions with random operands and operators guarantee no pre-training overlap. By comparing RL behavior on contaminated vs. clean benchmarks, the approach isolates genuine reasoning improvements from exploitation of memorization.
Architecture Overview
Contamination Detection Module: ROUGE-L scoring on partial-prompt completions to measure benchmark reconstruction ability
Partial-Prompt Test Framework: Truncate prompts to 60%, measure reconstruction and answer accuracy separately
RL Evaluation Pipeline: GRPO algorithm with three reward variants (correct, random, inverted) to probe memorization exploitation
Clean Benchmark Generator: Procedurally-generated arithmetic expressions ensuring zero data contamination
Activation Analysis: Similarity metrics comparing responses between contaminated and clean benchmarks using ROUGE-L
Signal Detection Method: Exploit-bias mechanism analysis via gradient tracing on high-probability tokens
Implementation
Contamination Detection via Partial-Prompt Reconstruction
Measure how well models reconstruct full benchmarks from incomplete prompts.
"""
Truncate problem to truncation_ratio of original length.
E.g., 60% means first 60% of text shown, model reconstructs rest.
Args:
problem_text: Full problem text
truncation_ratio: Fraction to keep (default 0.6 = 60%)
Returns:
truncated: Partial problem
full: Full problem for comparison
"""
int
len
' '
return
def
completion_reconstruction_rate
self, problems, truncation_ratio=0.6
"""
Measure ROUGE-L between model completions and ground truth.
High scores indicate memorization; low scores indicate reasoning.
Args:
problems: List of problem texts
truncation_ratio: How much of prompt to show
Returns:
mean_rouge: Average ROUGE-L score (0-1)
per_problem_scores: Individual ROUGE-L for each problem
"""
for
in
self
# Generate completion from truncated prompt
self
'pt'
with
self
len
self
50
0.7
0.9
self
0
# Score reconstruction quality
self
'rougeL'
return
sum
len
def
partial_prompt_answer_accuracy
self, qa_pairs, truncation_ratio=0.6
"""
Test if models recover correct answers from partial prompts.
Args:
qa_pairs: List of {'question': ..., 'answer': ...} dicts
truncation_ratio: How much of question to show
Returns:
accuracy: Fraction of correct answers recovered
"""
0
for
in
'question'
'answer'
self
self
'pt'
with
self
50
0.0
# Greedy for answer extraction
self
0
# Simple answer matching (for math: check if final number appears)
Generate arithmetic expressions with random operands, guaranteeing no pre-training contamination.
import random
from typing importList, TupleclassRandomCalculationDataset:
def__init__(self, seed=42, min_steps=1, max_steps=20):
random.seed(seed)
self.min_steps = min_steps
self.max_steps = max_steps
self.operators = ['+', '-', '*', '//'] # Integer operationsdefgenerate_expression(self, num_steps: int) -> Tuple[str, float]:
"""
Generate random arithmetic expression with num_steps operations.
Args:
num_steps: Number of computation steps (1-20)
Returns:
expression: String like "5 + 3 * 2 - 1"
answer: Computed result following operator precedence
"""# Start with random operand
operands = [random.randint(1, 100) for _ inrange(num_steps + 1)]
operators = [random.choice(self.operators) for _ inrange(num_steps)]
# Construct expression
expression = str(operands[0])
for op, operand inzip(operators, operands[1:]):
expression += f" {op}{operand}"# Evaluate with proper precedencetry:
answer = eval(expression)
except ZeroDivisionError:
# Retry if division by zeroreturnself.generate_expression(num_steps)
return expression, answer
defcreate_qa_pairs(self, num_pairs: int = 1000) -> List[dict]:
"""
Create QA dataset with guaranteed no pre-training contamination.
Args:
num_pairs: Number of QA pairs to generate
Returns:
qa_pairs: List of {'question': ..., 'answer': ...} dicts
"""
qa_pairs = []
for _ inrange(num_pairs):
num_steps = random.randint(self.min_steps, self.max_steps)
expression, answer = self.generate_expression(num_steps)
qa_pairs.append({
'question': f"Calculate: {expression}",
'answer': str(int(answer)),
'steps': num_steps
})
return qa_pairs
# Create clean dataset
generator = RandomCalculationDataset(min_steps=1, max_steps=20)
clean_benchmark = generator.create_qa_pairs(num_pairs=500)
print(f"Generated {len(clean_benchmark)} clean problems")
print(f"Example: {clean_benchmark[0]}")
# Example: {'question': 'Calculate: 47 + 12 * 3 - 5', 'answer': '78', 'steps': 3}
RL Training with Contamination Detection
Compare RL behavior on contaminated vs. clean benchmarks to isolate memorization exploitation.
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch.optim as optim
classRLBenchmarkComparison:
def__init__(self, model_name):
self.model = AutoModelForCausalLM.from_pretrained(model_name)
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
defcompute_reward(self, generated_answer, ground_truth, reward_type='correct'):
"""
Three reward variants to probe memorization exploitation.
Args:
generated_answer: Model's generated answer
ground_truth: Correct answer
reward_type: 'correct' (ground truth), 'random' (50% prob), 'inverted' (wrong reward)
Returns:
reward: Scalar reward value
"""if reward_type == 'correct':
# Standard reward: 1 if correct, 0 otherwisereturn1.0if generated_answer == ground_truth else0.0elif reward_type == 'random':
# Random reward: 50% chance of 1return1.0if random.random() < 0.5else0.0elif reward_type == 'inverted':
# Inverted reward: 1 if WRONGreturn0.0if generated_answer == ground_truth else1.0defevaluate_with_rewards(self, benchmark, reward_type='correct', num_samples=100):
"""
Train model briefly on benchmark with specified reward type.
Compare performance and answer similarity.
Args:
benchmark: QA dataset
reward_type: Which reward variant to use
num_samples: How many examples to use
Returns:
accuracy: Final accuracy on benchmark
response_similarity: ROUGE-L of generated responses
"""self.model.train()
optimizer = optim.AdamW([p for p inself.model.parameters() if p.requires_grad],
lr=5e-7)
responses = []
total_loss = 0.0for i, qa_pair inenumerate(benchmark[:num_samples]):
question = qa_pair['question']
answer = qa_pair['answer']
# Generate response
input_ids = self.tokenizer.encode(question, return_tensors='pt')
with torch.no_grad():
output_ids = self.model.generate(
input_ids,
max_new_tokens=20,
temperature=1.0
)
generated_text = self.tokenizer.decode(output_ids[0])
responses.append(generated_text)
# Compute reward
reward = self.compute_reward(generated_text, answer, reward_type)
# Simple reward-weighted loss (maximize reward)# In practice, use GRPO (Group Relative Policy Optimization)if reward > 0:
loss = -torch.tensor(reward, dtype=torch.float32)
else:
loss = torch.tensor(1.0, dtype=torch.float32)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
return {
'average_loss': total_loss / num_samples,
'responses': responses
}
# Compare on contaminated vs. clean benchmarks
contaminated = math_500_problems[:100]
clean = generator.create_qa_pairs(100)
print("Testing with CORRECT rewards:")
contaminated_result = model_eval.evaluate_with_rewards(contaminated, reward_type='correct')
clean_result = model_eval.evaluate_with_rewards(clean, reward_type='correct')
print(f" Contaminated: loss={contaminated_result['average_loss']:.4f}")
print(f" Clean: loss={clean_result['average_loss']:.4f}")
print("\nTesting with RANDOM rewards (should fail on clean):")
contaminated_result = model_eval.evaluate_with_rewards(contaminated, reward_type='random')
clean_result = model_eval.evaluate_with_rewards(clean, reward_type='random')
print(f" Contaminated: loss={contaminated_result['average_loss']:.4f}")
print(f" Clean: loss={clean_result['average_loss']:.4f}")
If model benefits from random/inverted rewards on contaminated but not clean, memorization is the issue.
Response Similarity Analysis
Measure ROUGE-L of model responses to detect memory vs. reasoning patterns.
from rouge_score import rouge_scorer
defanalyze_response_similarity(responses_contaminated, responses_clean):
"""
Compare response patterns: high ROUGE on contaminated + low on clean = memorization.
Args:
responses_contaminated: Model outputs on contaminated benchmark
responses_clean: Model outputs on clean benchmark
Returns:
contaminated_similarity: Average ROUGE-L between model responses
clean_similarity: Average ROUGE-L between model responses
"""
rouge = rouge_scorer.RougeScorer(['rougeL'], use_stemmer=True)
defcompute_similarity(responses):
iflen(responses) < 2:
return0.0
similarities = []
for i inrange(len(responses) - 1):
score = rouge.score(responses[i], responses[i+1])['rougeL'].fmeasure
similarities.append(score)
returnsum(similarities) / len(similarities)
contam_sim = compute_similarity(responses_contaminated)
clean_sim = compute_similarity(responses_clean)
print(f"Response similarity on contaminated: {contam_sim:.3f}")
print(f"Response similarity on clean: {clean_sim:.3f}")
if contam_sim > 2 * clean_sim:
print("MEMORIZATION DETECTED: Contaminated responses are too similar (retrieved, not generated)")
else:
print("Responses show genuine diversity (likely reasoning-based)")
return contam_sim, clean_sim
Practical Guidance
Key Hyperparameters
Parameter
Value
Notes
ROUGE Truncation
60% of prompt
Standard partial-prompt test; 40-60% range works
RL Learning Rate
5e-7
Critical; prevents reward exploitation without learning
Group Size
16 samples per prompt
For GRPO algorithm
Batch Size
128
Standard for RL fine-tuning
Temperature
1.0
Exploration; 0 during greedy answer extraction
Random Reward Probability
50%
γ = 0.5 in paper
Clean Dataset Size
500-1000 problems
Adequate for detection; more data = higher confidence
Epsilon (GRPO clipping)
0.20
Controls exploitation bias
When to Use
Validating RL training results on mathematical benchmarks before publication
Checking if language model improvements are genuine reasoning or memorization
Building evaluation frameworks that guarantee benchmark integrity
Designing datasets for models that may have seen training data origins
Comparing models where one might have contaminated pre-training
When NOT to Use
Real-time inference; contamination detection is offline validation
Evaluating models on completely unseen domains (clean dataset is necessary)
Scenarios where pre-training data is rigorously documented and available
Quick prototyping; this adds validation overhead
Common Pitfalls
Using single-prompt truncation: One truncation point is not representative; test at 40%, 50%, 60% to get robust signal
Ignoring answer extraction quality: Simple string matching misses near-correct answers; use semantic similarity for math
Conflating low ROUGE with reasoning: Low ROUGE-L (<10%) is necessary but not sufficient; verify with clean benchmark comparison
Insufficient clean dataset size: <100 problems is too small; generate at least 500 with varied difficulty for reliable signal
Applying uniform RL parameters: Models that memorize respond differently to learning rates; tune per-model and measure reward responsiveness
Forgetting control: models: Must test BOTH incorrect (random/inverted) AND correct rewards; correct rewards should improve on clean data
Reference
Qian, L., Chen, X., Zhang, Y., et al. (2024). Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv preprint arXiv:2507.10532.
