メニュー
Evaluate reasoning model capabilities by analyzing three complexity-dependent behavioral regimes and identifying fundamental limitations in symbolic manipulation rather than computational budgets.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill illusion-of-thinkingコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
SOC 職業分類に基づく
SKILL.md を表示中
| name | illusion-of-thinking |
| title | The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.06941 |
| keywords | ["reasoning models","problem complexity","thinking traces","generalization failure"] |
| description | Evaluate reasoning model capabilities by analyzing three complexity-dependent behavioral regimes and identifying fundamental limitations in symbolic manipulation rather than computational budgets. |
Large Reasoning Models (LRMs) demonstrate impressive benchmark performance, but this research challenges whether improvements reflect genuine reasoning or sophisticated pattern matching. Using algorithmically-structured puzzles with controllable difficulty, the authors reveal three complexity-dependent behavioral regimes where models fundamentally fail to generalize beyond training distributions.
Create puzzle environments with adjustable parameters:
class PuzzleEnvironment:
def __init__(self, puzzle_type: str, complexity_level: int):
self.puzzle_type = puzzle_type
self.complexity = complexity_level
self.state = self.initialize_state()
def initialize_state(self) -> dict:
"""Generate puzzle state with controlled difficulty."""
if self.puzzle_type == "tower_of_hanoi":
return {"disks": self.complexity, "pegs": [[], [], []]}
elif self.puzzle_type == "checker_jumping":
{: ._create_board(.complexity)}
() -> :
._apply_transition(action)
() -> :
._compute_lower_bound()
Analyze reasoning behavior across complexity regimes:
class ThinkingAnalyzer:
def extract_trace_features(self, thinking_text: str,
final_answer: str) -> dict:
"""Extract patterns from extended reasoning traces."""
features = {
"token_count": len(thinking_text.split()),
"solution_discovery_point": self._find_first_solution(thinking_text),
"error_recovery_attempts": self._count_recoveries(thinking_text),
"fixation_patterns": self._detect_fixation(thinking_text),
"correctness": self._validate_answer(final_answer)
}
return features
def identify_regime(self, features: dict,
complexity: int) -> str:
"""Classify behavior into three regimes."""
if features["correctness"] and complexity < self.low_threshold:
return "LOW_COMPLEXITY"
elif features["correctness"] and complexity < self.medium_threshold:
return "MEDIUM_COMPLEXITY"
else:
return "HIGH_COMPLEXITY_COLLAPSE"
Test reasoning vs non-reasoning models under equivalent budgets:
def run_comparative_test(model_reasoning, model_baseline,
puzzle_env: PuzzleEnvironment,
max_tokens: int = 512) -> dict:
"""Compare performance with matched computational budgets."""
# Get reasoning model output
reasoning_output = model_reasoning.generate(
puzzle_env.to_prompt(),
max_tokens=max_tokens,
thinking_budget=max_tokens // 2
)
# Get baseline output with equivalent token budget
baseline_output = model_baseline.generate(
puzzle_env.to_prompt(),
max_tokens=max_tokens
)
return {
"reasoning_correct": validate_solution(reasoning_output),
"baseline_correct": validate_solution(baseline_output),
"reasoning_tokens": reasoning_output.token_count,
"reasoning_effort_trajectory": extract_effort_curve(
reasoning_output.thinking_trace
)
}
Key Finding: Models allocate more thinking initially, then counterintuitively reduce effort approaching collapse points—despite adequate remaining budget. This suggests the failure is not computational but rooted in fundamental symbolic manipulation limitations.
Experimental Design Tips:
When to Apply: Use this framework to rigorously evaluate claims about reasoning capability improvements, especially when benchmarks may suffer from data leakage or distribution shift.
The work employs three behavioral regimes:
The "reasoning effort paradox" indicates models reduce thinking as problems become intractable—counterintuitive behavior suggesting training dynamics rather than genuine search limitations.