菜单
Improve reasoning models by aligning their meta-predictions with actual rollouts through self-generated training signals. Trigger: accelerate reasoning model training while maintaining performance through better meta-cognitive awareness.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill meta-aware-reasoning命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | meta-aware-reasoning |
| title | Meta-Awareness Enhances Reasoning: Self-Alignment via MASA |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.03259 |
| keywords | ["meta-awareness","reasoning-models","self-alignment","rl-training","computational-efficiency"] |
| description | Improve reasoning models by aligning their meta-predictions with actual rollouts through self-generated training signals. Trigger: accelerate reasoning model training while maintaining performance through better meta-cognitive awareness. |
Reasoning models like o1 can predict whether they'll find solutions (meta-awareness), but this metacognitive ability is often misaligned with actual problem-solving success. MASA (Meta-Awareness via Self-Alignment) trains models to align meta-predictions with reality by using self-generated rollouts as training signals. This yields 1.28x training speedup, 19.3% accuracy gain on AIME25, and enables early stopping of unpromising reasoning paths.
The key insight: A model's own rollouts reveal the ground truth about its reasoning—use these self-generated labels to refine meta-cognition without external supervision.
Model learns to predict "will I solve this?" based on partial reasoning.
class MetaPredictionModule:
"""
Train model to predict solution success probability.
"""
def __init__(self, model):
self.model = model
def extract_meta_prediction(self, partial_trace):
"""
Get model's prediction: "Given reasoning so far, likely to solve?"
Args:
partial_trace: Reasoning text generated so far
Returns:
Probability estimate [0, 1]
"""
prompt = (
f"Given this reasoning so far:\n{partial_trace}\n\n"
f"What's the probability I'll solve this? Answer: [0-100]%"
)
response = self.model.generate(prompt, max_tokens=10)
# Extract number
prob_str = extract_number(response)
probability = (prob_str) /
probability
():
prompt =
logits = .model.get_logits(prompt)
prediction_token_id = .model.predict_next_token(logits)
log_prob = torch.log_softmax(logits, dim=-)[prediction_token_id]
log_prob
Use actual solution success to create ground-truth labels for meta-predictions.
class SelfSupervisedMetaTrainer:
"""
Train meta-prediction using model's own rollout outcomes.
"""
def __init__(self, model):
self.model = model
def create_meta_training_pair(self, problem, max_thinking_tokens=4096):
"""
Generate partial reasoning and check eventual success.
Args:
problem: Problem statement
max_thinking_tokens: Budget for reasoning
Returns:
(partial_trace, meta_prediction, actual_success)
"""
# Run full reasoning
full_trace = self.model.generate(
problem,
max_tokens=max_thinking_tokens
)
# Extract solution
full_solution = extract_solution(full_trace)
actual_success = evaluate_correctness(full_solution, problem)
# Get meta-prediction at intermediate point (50% through reasoning)
midpoint = len(full_trace) // 2
partial_trace = full_trace[:midpoint]
meta_pred_prob = self.extract_meta_prediction(partial_trace)
return {
"partial_trace": partial_trace,
"meta_prediction": meta_pred_prob,
"actual_success": actual_success,
"full_trace": full_trace
}
def filter_trivial_cases(self, training_pairs):
"""
Remove zero-variance problems (always solved or never solved).
These don't help meta-awareness training.
Returns:
Filtered pairs with meaningful variance
"""
filtered = []
for pair in training_pairs:
# Only keep examples where meta-prediction could differ from outcome
meta_pred = pair["meta_prediction"]
actual = pair["actual_success"]
# Variance exists if prediction differs from outcome
if (meta_pred > 0.5 and not actual) or (meta_pred < 0.5 and actual):
filtered.append(pair)
elif 0.3 < meta_pred < 0.7: # Uncertain predictions are valuable
filtered.append(pair)
return filtered
Create loss that aligns predictions with actual outcomes.
def compute_meta_alignment_loss(training_pair):
"""
Loss for aligning meta-prediction with ground-truth outcome.
Key insight: Use actual success as supervision for meta-prediction.
"""
meta_pred = training_pair["meta_prediction"]
actual_success = float(training_pair["actual_success"])
# Cross-entropy loss: treat as binary classification
# Model predicts probability; ground truth is 0 or 1
epsilon = 1e-7
loss = -(
actual_success * torch.log(meta_pred + epsilon) +
(1 - actual_success) * torch.log(1 - meta_pred + epsilon)
)
return loss
Use meta-predictions to halt unpromising reasoning paths.
class EarlyStoppingController:
"""
Use meta-awareness to decide when to stop reasoning.
"""
def __init__(self, model, threshold=0.1):
self.model = model
self.threshold = threshold
def should_continue_reasoning(self, partial_trace):
"""
Decide whether to continue generating reasoning tokens.
Args:
partial_trace: Reasoning generated so far
Returns:
Boolean: continue or stop
"""
meta_prob = self.extract_meta_prediction(partial_trace)
# Stop if model thinks solution unlikely
if meta_prob < self.threshold:
return False
return True
def generate_with_early_stopping(self, problem, max_tokens=4096):
"""
Generate reasoning, stopping early if unlikely to succeed.
"""
trace = ""
for _ in range(max_tokens):
# Generate one token
next_token = self.model.generate_one_token(problem + trace)
trace += next_token
# Check meta-awareness every 100 tokens
if len(trace.split()) % 100 == 0:
if not self.should_continue_reasoning(trace):
# Early stop: backtrack and try different approach
trace = trace[:len(trace) // 2] # Reset to midpoint
trace += "\n[Alternative approach]\n"
# Terminal condition
if is_complete_solution(trace):
break
return trace
Integrate meta-training with main reasoning model training.
def train_with_masa(
model,
dataset,
config
):
"""
Train reasoning model with meta-awareness self-alignment.
"""
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
meta_trainer = SelfSupervisedMetaTrainer(model)
early_stopper = EarlyStoppingController(model, threshold=0.15)
for epoch in range(config.num_epochs):
epoch_loss = 0
num_meta_examples = 0
for problem_id, problem in enumerate(dataset):
# Standard reasoning: generate solution
reasoning = model.generate(problem, max_tokens=2048)
solution = extract_solution(reasoning)
is_correct = evaluate_correctness(solution, problem)
# Standard RL loss
log_prob = model.compute_log_prob(reasoning)
reward = 1.0 if is_correct else -1.0
rl_loss = -reward * log_prob
# Meta-awareness training: self-supervised
# Create training pair from this rollout
meta_pair = meta_trainer.create_meta_training_pair(problem)
# Filter trivial cases
if not is_trivial_case(meta_pair):
# Meta-alignment loss
meta_loss = compute_meta_alignment_loss(meta_pair)
# Combined loss
total_loss = rl_loss + 0.5 * meta_loss
num_meta_examples += 1
else:
total_loss = rl_loss
# Update
optimizer.zero_grad()
total_loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
epoch_loss += total_loss.item()
if (problem_id + 1) % 100 == 0:
print(f"Epoch {epoch}, Problem {problem_id}: "
f"loss={epoch_loss / (problem_id + 1):.4f}, "
f"meta_examples={num_meta_examples}")
return model
Measure speedup and accuracy gains from meta-awareness.
def evaluate_masa_training(model, benchmark_dataset):
"""
Benchmark accuracy and computational efficiency.
"""
results = {
"accuracy": 0,
"avg_tokens_generated": 0,
"early_stop_rate": 0
}
correct = 0
total_tokens = 0
early_stops = 0
early_stopper = EarlyStoppingController(model, threshold=0.15)
for problem in benchmark_dataset:
# Generate with early stopping
reasoning = early_stopper.generate_with_early_stopping(
problem,
max_tokens=4096
)
solution = extract_solution(reasoning)
if evaluate_correctness(solution, problem):
correct += 1
total_tokens += len(reasoning.split())
# Track early stops
if len(reasoning.split()) < 2048: # Stopped before max
early_stops += 1
results["accuracy"] = correct / len(benchmark_dataset) * 100
results["avg_tokens_generated"] = total_tokens / len(benchmark_dataset)
results["early_stop_rate"] = early_stops / len(benchmark_dataset) * 100
print(f"Accuracy: {results['accuracy']:.1f}%")
print(f"Avg tokens: {results['avg_tokens_generated']:.0f}")
print(f"Early stop rate: {results['early_stop_rate']:.1f}%")
return results
Hyperparameters:
When to Use:
When NOT to Use:
Meta-Awareness Enhances Reasoning: Self-Alignment via Reinforcement Learning — arXiv:2510.03259