rest-stress-testing-reasoning

name	rest-stress-testing-reasoning
title	REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.10541
keywords	["Model Evaluation","Stress Testing","Reasoning Models","Benchmark Design","Multi-Problem Prompts"]
description	Evaluate large reasoning models under stress by asking multiple problems simultaneously, revealing true multi-tasking capacity and context management. Use to identify robustness gaps invisible in single-problem evaluations and discriminate model capabilities beyond traditional benchmarks.

REST: Stress Testing Reasoning Models Through Multi-Problem Evaluation

State-of-the-art reasoning models like DeepSeek-R1 achieve near-perfect scores on standard benchmarks, yet this masks a critical weakness: performance degrades significantly when answering multiple questions simultaneously. REST (Reasoning Extended Stress Testing) transforms existing benchmarks by concatenating multiple problems into single prompts, measuring how models handle reasoning under load. Results reveal that even top models experience 20-40% accuracy drops under stress, challenging assumptions that LLMs inherently function as multi-problem solvers.

The key insight is that benchmark scores measure single-problem performance; real-world applications require simultaneous reasoning over multiple tasks. By systematically increasing stress level (number of concurrent problems), REST discriminates models that truly maintain reasoning clarity from those that degrade under cognitive load.

Core Concept

REST transforms any benchmark by creating stress-level-s prompts containing s consecutive questions concatenated into a single prompt. Each original problem appears exactly s times across the full prompt set but at different positions to prevent positional bias. This reveals three critical capabilities: context management (handling multiple reasoning threads), answer localization (retrieving answers to specific questions), and focus maintenance (preventing interference between problems).

The evaluation framework captures three phenomena: global performance degradation with stress level, position effects (earlier problems tend to score higher), and enhanced discrimination between models that appear similar on single-problem benchmarks.

Architecture Overview

• Benchmark Transformation Engine: Converts standard benchmarks into multi-problem stress variants while controlling positional bias
• Stress Level Controller: Parametrizes number of concurrent problems (s=1 to s=12 depending on task difficulty)
• Answer Extraction Module: Handles both rule-based extraction (for marked formats like LaTeX \boxed{}) and LLM-based extraction (for unstructured answers)
• Position Bias Mitigation: Rotates question positions across prompt variants to isolate position effects from model capability
• Multi-Model Evaluation Framework: Tests 34 models across 7 benchmarks with consistent hyperparameter application

Implementation

Benchmark Transformation: Creating Stress-Level Prompts

Convert standard single-problem benchmarks into multi-problem stress variants.

from typing import List, 
 random

 :
    

     ():
        
        .problems = problems
        .num_problems = (problems)
        random.seed(seed)

     () -> [[, []]]:
        
        stress_prompts = []

        
         i  (.num_problems):
            
            problem_indices = [
                (i + j) % .num_problems
                 j  (stress_level)
            ]

            
            prompt_parts = []
            ground_truths = []

             q_num, prob_idx  (problem_indices):
                problem = .problems[prob_idx]

                
                question_text = 
                prompt_parts.append(question_text)
                ground_truths.append(problem[])

            
            prompt = .join(prompt_parts)

            stress_prompts.append((prompt, ground_truths))

         stress_prompts

     () -> :
        
        positions = (accuracy_by_position.keys())
        accuracies = [accuracy_by_position[p]  p  positions]

        
        position_coeffs = (((positions)))
        mean_x = (position_coeffs) / (position_coeffs)
        mean_y = (accuracies) / (accuracies)

        numerator = ((x - mean_x) * (y - mean_y)  x, y  (position_coeffs, accuracies))
        denominator = ((x - mean_x) **   x  position_coeffs)

        slope = numerator / denominator  denominator >   

         {
            : accuracy_by_position,
            : slope,
            : accuracies[],
            : accuracies[-],
            : slope < -  
        }


math_problems = [
    {: , : },
    {: , : },
    
]

transformer = RESTBenchmarkTransformer(math_problems)


stress_1_prompts = transformer.create_stress_prompts(stress_level=)  
stress_3_prompts = transformer.create_stress_prompts(stress_level=)  
stress_5_prompts = transformer.create_stress_prompts(stress_level=)  

()
()
()
()
(stress_3_prompts[][])