Scale model performance at test time by generating multiple reasoning trajectories and selecting the best using a self-supervised process reward model. MetaStone-S1 achieves 32B-equivalent performance using only 32B parameters and 53M for trajectory scoring, learning process rewards from outcome labels alone without process annotations.
Scale model performance at test time by generating multiple reasoning trajectories and selecting the best using a self-supervised process reward model. MetaStone-S1 achieves 32B-equivalent performance using only 32B parameters and 53M for trajectory scoring, learning process rewards from outcome labels alone without process annotations.
Test-Time Scaling with Reflective Generative Models: More Compute at Inference
Inference-time compute is often cheaper than training-time compute, yet most models use fixed generation sequences. Test-time scaling generates multiple candidate trajectories and selects the highest-confidence path, dramatically improving reasoning performance without additional training. The challenge is scoring trajectories—process reward models typically require expensive per-step annotations. Reflective Generative Models solve this with a unified backbone that shares parameters between generation and evaluation, plus a self-supervised process reward model that learns trajectory scoring from only final-answer correctness.
The key insight is that a single shared backbone can simultaneously generate reasoning steps and evaluate them. By training a process reward model to distinguish correct from incorrect reasoning using only outcome labels, you eliminate annotation bottlenecks while achieving performance parity with o3-mini using 32B parameters instead of much larger models.
Core Concept
Test-time scaling combines three components:
Reflective Generation: A single backbone generates reasoning trajectories while simultaneously serving as a process evaluator via additional scoring heads
Self-Supervised Process Reward Model (SPRM): Learns to score reasoning steps using only binary outcome supervision (correct/incorrect final answer)
Multi-Trajectory Selection: Generate k trajectories at test time (k=2, 8, 32 for low/medium/high compute), select highest-scoring path using SPRM
The model reaches an "Aha Moment" during training where it transitions from treating all reasoning patterns identically to meaningfully discriminating good trajectories from bad ones, using only final answer signals.
Architecture Overview
Shared Backbone: LLM generating both reasoning tokens and trajectory scores (e.g., Llama-based 32B)
Generation Head: Standard language modeling head for producing reasoning text
model: ReflectiveGenerativeModel,
input_ids: torch.Tensor,
num_trajectories: int = 8,
max_length: int = 512,
temperature: float = 1.0
Tuple
List
List
"""
Generate k reasoning trajectories and score them.
Args:
model: Reflective generative model
input_ids: Problem prompt (batch_size, seq_len)
num_trajectories: Number of trajectories to generate per input
max_length: Maximum trajectory length
temperature: Sampling temperature (higher = more diverse)
Returns:
trajectories: List of (batch, seq_len) token sequences
trajectory_scores: List of (batch,) scalar scores
"""
"""
Single training step combining generation and process reward learning.
Args:
model: Reflective generative model
input_ids: Problem prompts (batch, seq_len)
target_ids: Correct reasoning trajectories (batch, target_len)
is_correct: Outcome correctness (batch,)
optimizer: Training optimizer
alpha: Weighting of SPRM loss vs generation loss
"""
# Forward pass
# Generation loss (standard language modeling)
1
1
1
1
1
# Self-supervised process reward loss
# Combined loss
return
This implementation shows the core reflective architecture: shared backbone for generation and scoring, plus self-supervised trajectory learning from outcome labels.
Practical Guidance
Aspect
Recommended Value
Notes
Trajectory Count (k)
2 (low), 8 (medium), 32 (high)
8× compute multiplier per increment
Temperature
0.8-1.2 for generation
Higher = more diverse trajectories
SPRM Loss Weight (α)
0.3-0.5
Balance generation vs. trajectory discrimination
Confidence Threshold
0.2-0.4 (score variance)
Filter noisy samples during SPRM training
Aha Moment Detection
Monitor SPRM accuracy
Should jump 30-50% at transition phase
Batch Size
128-256 during trajectory generation
Memory-intensive due to k×batch size tokens
When to Use Test-Time Scaling
Latency-tolerant inference: Can afford 2-32× longer generation (seconds, not milliseconds)
Cost-conscious scaling: Better cost/accuracy tradeoff than training larger models
Adaptive compute: Select k=2 for simple queries, k=32 for complex ones
Ensemble learning: Multiple trajectories provide uncertainty estimates
Reward hacking defense: Process rewards reduce vulnerability to superficial outputs
When NOT to Use
Real-time interactive systems: 32× generation slowdown unacceptable (>5 second latency)
Models already optimized for reasoning: O1/O3 families already use advanced test-time scaling
Supervised process rewards available: If you have per-step annotations, use supervised process rewards (higher quality)
Vocabulary <5K or >100K: Self-supervised SPRM assumes diverse token distributions; extreme vocabularies cause training instability
Very short reasoning tasks (1-2 steps): Trajectory diversity negligible; additional compute doesn't help
Common Pitfalls
Temperature Too Low: Setting temperature <0.5 creates near-identical trajectories. Increase to 1.0-1.5 for diversity.
Aha Moment Missed: If SPRM loss doesn't show sharp accuracy transition, learning rate is too high (causes oscillation) or too low (model doesn't learn). Use lr=1e-4 with 10% warmup.
Confidence Weighting Ineffective: If you filter >50% of samples, threshold is too aggressive. Reduce confidence_threshold to 0.1.
Trajectory Score Collapse: All trajectories score similarly = model hasn't learned discrimination. Add auxiliary losses (e.g., contrastive learning between correct/incorrect pairs).
Ignoring Output Token Limits: Long-sequence generation with k=32 rapidly exhausts memory. Implement sliding-window scoring or token budget constraints.
Reference
Ying, L., Wang, R., et al. (2025). Test-Time Scaling with Reflective Generative Model. arXiv preprint arXiv:2507.01951.
