illusion-of-thinking

name	illusion-of-thinking
title	The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.06941
keywords	["reasoning models","problem complexity","thinking traces","generalization failure"]
description	Evaluate reasoning model capabilities by analyzing three complexity-dependent behavioral regimes and identifying fundamental limitations in symbolic manipulation rather than computational budgets.

The Illusion of Thinking

Core Concept

Large Reasoning Models (LRMs) demonstrate impressive benchmark performance, but this research challenges whether improvements reflect genuine reasoning or sophisticated pattern matching. Using algorithmically-structured puzzles with controllable difficulty, the authors reveal three complexity-dependent behavioral regimes where models fundamentally fail to generalize beyond training distributions.

Architecture Overview

• Four puzzle environments: Tower of Hanoi, Checker Jumping, River Crossing, Blocks World enable fine-grained complexity control
• Reasoning trace analysis: Extract intermediate solutions and thinking patterns from extended traces
• Comparative evaluation: Match reasoning/non-reasoning model pairs with equivalent inference compute budgets
• Sequential validation: Custom simulators verify each move step-by-step

Implementation

Step 1: Design Controllable Puzzle Environment

Create puzzle environments with adjustable parameters:

class PuzzleEnvironment:
    def __init__(self, puzzle_type: str, complexity_level: int):
        self.puzzle_type = puzzle_type
        self.complexity = complexity_level
        self.state = self.initialize_state()

    def initialize_state(self) -> dict:
        """Generate puzzle state with controlled difficulty."""
        if self.puzzle_type == "tower_of_hanoi":
            return {"disks": self.complexity, "pegs": [[], [], []]}
        elif self.puzzle_type == "checker_jumping":
             {: ._create_board(.complexity)}

     () -> :
        
         ._apply_transition(action)   

     () -> :
        
         ._compute_lower_bound()