Build process reward models for tabular reasoning by grounding intermediate reasoning steps in executable tool operations. Train on 60K+ step-level annotations combining verification rationales with tool execution feedback. TaTToo achieves 30.9% improvement over policies using only final rewards, outperforming 72B reasoning models with just 8B parameters via better step-level supervision.
Build process reward models for tabular reasoning by grounding intermediate reasoning steps in executable tool operations. Train on 60K+ step-level annotations combining verification rationales with tool execution feedback. TaTToo achieves 30.9% improvement over policies using only final rewards, outperforming 72B reasoning models with just 8B parameters via better step-level supervision.
TaTToo: Tool-Grounded Thinking PRM for Tabular Reasoning
Core Concept
Process Reward Models (PRMs) enable test-time scaling via intermediate step-level feedback, but existing PRMs struggle with table-specific operations (sub-table retrieval, schema navigation). TaTToo grounds each reasoning step in executable tool operations—SQL queries, table transforms, schema checks—generating precise reward signals for what constitutes progress in data analysis tasks.
Architecture Overview
Dual-Stage Training: (1) Supervised fine-tuning on verification rationales; (2) reinforcement learning with tool-grounded rewards
Tool-Based Verification: Each step grounds in actual tool execution (pandas operations, SQL queries), not just text descriptions
60K+ Step Annotations: Scalable pipeline generates high-quality step-level training data from tool execution traces
Reward Shaping: Reward models learn to evaluate reasoning quality based on tool-execution feasibility and correctness
Generalization: Trained TaTToo transfers across benchmarks and test-time scaling strategies
Implementation Steps
1. Grounding Reasoning Steps in Tool Operations
Each reasoning step must map to executable tool operations for verification.
2. Data Generation Pipeline: 60K+ Step Annotations
Automated pipeline generating training data from tool execution traces.
defgenerate_step_level_annotations(questions, tables, max_traces=10000):
"""
Generate 60K+ step-level training examples from tabular reasoning traces.
"""
training_examples = []
for question, table inzip(questions, tables):
# Generate multiple reasoning traces per questionfor trace_idx inrange(5): # 5 traces per question
trace = generate_reasoning_trace(question, table)
# Verify each step
verification_result = trace.verify_full_trace()
# Extract step-level training examplesfor step_idx, step inenumerate(trace.steps):
# Determine if step is on "good path" (leads to correct answer)
is_on_correct_path = all(
trace.steps[i].verification_status
for i inrange(step_idx + 1)
) and verification_result['final_correct']
example = {
'question': question,
'table': table,
'step_number': step_idx,
'reasoning': step.reasoning,
'tool_operation': step.operation,
'step_correct': step.verification_status,
'on_correct_path': is_on_correct_path,
'intermediate_output': step.actual_output,
'label': 1.0if is_on_correct_path else0.0
}
training_examples.append(example)
return training_examples
# Example: SQL-grounded tabular reasoning
trace_example = TabularReasoningTrace(
question="How many employees have salary > 100k and joined after 2020?",
table="employee_table",
expected_answer=42
)
trace_example.add_step(
reasoning="First, filter employees with salary > 100k",
tool_op="SELECT * FROM employee_table WHERE salary > 100000",
expected_output="[150000 rows]"
)
trace_example.add_step(
reasoning="Filter those who joined after 2020",
tool_op="SELECT * FROM filtered_above WHERE join_date > '2020-12-31'",
expected_output="[42 rows]"
)
trace_example.add_step(
reasoning="Count the results",
tool_op="SELECT COUNT(*) FROM filtered_2020_above",
expected_output=42
)
3. Dual-Stage Training: SFT + RL
Stage 1: Supervised fine-tuning on verification rationales. Stage 2: RL with tool-grounded rewards.
classTaTTooTrainer:
def__init__(self, base_model='qwen-8b'):
self.model = base_model
self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5)
defstage1_supervised_finetuning(self, training_examples, num_epochs=3):
"""
SFT on step-level annotations: learn to evaluate step quality.
"""for epoch inrange(num_epochs):
total_loss = 0for example in training_examples:
# Input: question + table + step reasoning
input_text = f"""Question: {example['question']}
Table: {example['table']}
Step {example['step_number']}: {example['reasoning']}
Is this step correct? (Binary classification)"""# Target: label (0 or 1)
target = example['label']
# Forward pass
logits = self.model.predict(input_text)
loss = torch.nn.functional.binary_cross_entropy_with_logits(logits, target)
# Backwardself.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
total_loss += loss.item()
print(f"Stage 1 Epoch {epoch}: Loss={total_loss/len(training_examples):.4f}")
defstage2_reinforcement_learning(self, eval_traces, num_steps=5000):
"""
RL: reward model learns from tool execution feedback.
Policy (reasoning model) learns to maximize expected rewards.
"""
policy = self.model # PRM acts as reward signal for policyfor step inrange(num_steps):
# Sample trace
trace = random.choice(eval_traces)
# Generate reasoning step
step_reasoning = trace.steps[random.randint(0, len(trace.steps) - 1)]
# Execute tool to get verification
step_reasoning.verify()
# Compute reward
tool_execution_successful = step_reasoning.verification_status
on_correct_path = all(s.verify() for s in trace.steps)
reward = 1.0if (tool_execution_successful and on_correct_path) else0.0# Policy gradient update
log_prob = self.model.log_probability(step_reasoning.reasoning)
loss = -log_prob * reward
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
if (step + 1) % 100 == 0:
print(f"Stage 2 Step {step+1}: Avg reward tracking...")
deftrain_tattoo(self, train_examples, eval_traces):
"""Full training pipeline."""print("Stage 1: Supervised fine-tuning...")
self.stage1_supervised_finetuning(train_examples)
print("Stage 2: Reinforcement learning...")
self.stage2_reinforcement_learning(eval_traces)
returnself.model
4. Evaluation on Tabular Benchmarks
Test-time scaling with TaTToo-trained reward models.
defevaluate_with_prm(policy, prm_reward_model, test_examples, num_beams=5):
"""
Use PRM for test-time scaling via beam search.
At each step, select highest-reward continuations.
"""
correct = 0for example in test_examples:
question = example['question']
table = example['table']
# Beam search with PRM guidance
beams = [(question, [])] # (current_state, step_history)for step_num inrange(20): # Max 20 steps
scored_beams = []
for state, history in beams:
# Generate next reasoning step
next_step = policy.generate_step(state, table, history)
# Score with PRM
reward = prm_reward_model.score(
question=question,
table=table,
step_number=step_num,
reasoning=next_step
)
scored_beams.append((reward, state + next_step, history + [next_step]))
# Keep top-k beams
scored_beams.sort(reverse=True)
beams = [(s, h) for _, s, h in scored_beams[:num_beams]]
# Evaluate final answer
final_answer = extract_answer_from_reasoning(beams[0][0])
if final_answer == example['expected_answer']:
correct += 1
accuracy = correct / len(test_examples)
return accuracy
# Benchmark results
results = {
'without_prm': {'accuracy': 0.45}, # Policy alone'with_tattoo_prm': {'accuracy': 0.55}, # +30.9% improvement'baseline_72b': {'accuracy': 0.52}, # Larger model without PRM
}
Practical Guidance
Tool Grounding: Every reasoning step must map to an executable operation. Abstract reasoning without tool execution doesn't generate reliable reward signals.
Training Data Scale: 60K+ examples provides good coverage across table schemas and operation types. Collect diverse traces (SQL, pandas, Excel operations) to ensure generalization.
Reward Shaping: Combine step-level correctness (tool execution) with trajectory-level signal (on-correct-path) to balance immediate feedback with long-horizon alignment.
Generalization: Train on 5 benchmarks, evaluate on held-out tasks. TaTToo generalizes better than task-specific PRMs due to tool-based grounding.
When to Use / When NOT to Use
Use When:
Training agents on data analysis tasks (SQL, pandas, spreadsheets)
Verifiable intermediate steps are available (query results, transformed tables)
Test-time scaling via beam search or planning is feasible
Smaller models need to match larger model performance
NOT For:
Tasks without executable verification (open-ended reasoning)
Domains where intermediate steps lack ground truth
Real-time low-latency inference (beam search is expensive)
Reference
This skill synthesizes findings from "TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning" (arXiv:2510.06217). Tool grounding enables reliable step-level reward signals for process-based RL in data analysis.
