art-of-scaling-test-time

name	art-of-scaling-test-time
title	The Art of Scaling Test-Time Compute for Large Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.02008
keywords	["test-time-scaling","llm-inference","majority-voting","beam-search","model-aware-strategies"]
description	Comprehensive empirical study recommending model-specific test-time scaling strategies (majority voting, first-finish search) across eight LLMs based on architectural family, problem difficulty, and compute budget rather than universal approaches.

Summary

The Art of Scaling Test-Time Compute is an empirical study and decision framework comparing test-time scaling (TTS) strategies across eight LLMs and four reasoning datasets, generating over 30 billion tokens. The key finding is that no universal TTS strategy dominates—optimal performance depends on model family, problem difficulty, and compute budget. The authors provide a practical recipe table for strategy selection.

Core Technique

Two Model Categories: The study identifies distinct categories based on post-training algorithms:

• Short-horizon models: Trained with algorithms other than GRPO (e.g., GSPO), show decreasing quality with longer traces
• Long-horizon models: Trained with GRPO, maintain or improve quality with longer traces

This distinction is critical for strategy selection.

TTS Strategy Comparison:

1. Beam Search: Keep top-k candidates at each step, select highest-reward final output
2. Majority Voting: Generate multiple independent solutions, return most common answer
3. First-Finish Search: Return first correct solution found, stop early
4. Last-Finish Search: Exhaust budget finding all correct solutions, return last found

Implementation

Model categorization: Profile your model on a small validation set:

1. Generate 10 solutions each of length L and 2L
2. Compare quality at both lengths
3. If quality improves at 2L: long-horizon; if decreases: short-horizon

Strategy selection: Use decision tree:

if problem_difficulty == "easy":
    use majority_voting(n_samples=3)
elif model_family == "long-horizon":
    use first_finish_search(budget=large)
else:  # short-horizon, hard problem
    use majority_voting(n_samples=5)

Budget allocation: Allocate compute proportionally:

if compute_budget < 2x:
    majority_voting(n=2)
elif compute_budget < 4x:
    first_finish_search(budget=4x)
else:
    first_finish_search(budget=compute_budget)

When to Use

• Deploying LLMs where reasoning quality significantly impacts downstream tasks
• Scenarios with variable compute budgets where you need adaptive scaling
• Applications requiring understanding which strategy is optimal for your model