Enable open-source reasoning model development with a 100K-instance Long CoT Collection, scaling from 1K o1 seed samples through guided synthesis with GPT-4o, achieving 2-3× RL performance gains.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Enable open-source reasoning model development with a 100K-instance Long CoT Collection, scaling from 1K o1 seed samples through guided synthesis with GPT-4o, achieving 2-3× RL performance gains.
One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL
Core Concept
Open-source reasoning model development faces a critical bottleneck: the "cold-start problem." High-quality long chain-of-thought data typically comes from proprietary models like OpenAI's o1, but researchers can't train models on private outputs. This paper demonstrates that short-CoT LLMs can generate high-quality long reasoning when guided by learned patterns. The approach creates a 100K-instance Long CoT Collection by (1) manually collecting 1K seed instances from o1, (2) scaling to 100K through GPT-4o synthesis guided by reasoning flows, and (3) filtering for correctness. Models initialized on this dataset achieve 2-3× greater RL performance gains compared to baselines, enabling independent reasoning model development without reliance on proprietary models.
Thought Budget Control: Manages reasoning token allocation to prevent overthinking
Open-Source Synthesis: Uses GPT-4o for guidance but not as data source
100K Scale Dataset: Sufficient for RL initialization with strong performance gains
Domain Generalization: Works beyond math (demonstrated on GPQA, multimodal tasks)
Implementation
Seed Dataset Creation: Manual collection of high-quality long CoT from o1
defcreate_seed_dataset(num_seed_samples=1000):
"""
Manually collect high-quality long CoT demonstrations.
Use o1 outputs as seed reasoning patterns.
"""
seed_dataset = []
for i inrange(num_seed_samples):
# Sample from diverse problem domains
problem = select_problem_from_o1(i)
# Get o1 reasoning (via API or cached results)
o1_response = query_o1_api(problem)
# Extract reasoning flow (high-level structure)
reasoning_flow = extract_reasoning_flow(o1_response)
# Extract thought budget (token allocation info)
thought_budget = estimate_thought_budget(o1_response)
seed_instance = {
: ,
: problem,
: o1_response[],
: o1_response[],
: reasoning_flow,
: thought_budget
}
seed_dataset.append(seed_instance)
seed_dataset
():
flow_prompt =
flow = call_gpt4o(flow_prompt)
flow
# Store seed instance
'id'
f'seed_{i}'
'problem'
'o1_reasoning'
'thinking'
'solution'
'answer'
'reasoning_flow'
'thought_budget'
return
def
extract_reasoning_flow
o1_response
"""
Extract high-level reasoning structure from o1 output.
Captures step-by-step outline without detailed explanations.
"""
f"""
Summarize the reasoning approach in 5-10 steps:
Thinking: {o1_response['thinking'][:2000]}
Return only the step outline.
"""
# Use GPT-4o to extract structure
return
Scaling Phase: Guided Synthesis: Generate 100K instances using retrieved demonstrations
defscale_to_100k_samples(seed_dataset, target_problems, scale_factor=100):
"""
Scale seed dataset to 100K by generating reasoning for new problems.
Use seed reasoning flows as guides, synthesize with GPT-4o.
"""
full_dataset = seed_dataset.copy()
for problem_id, problem inenumerate(target_problems):
iflen(full_dataset) >= 100000:
break# Step 1: Retrieve similar seed instances by embedding similarity
similar_seeds = retrieve_similar_seeds(problem, seed_dataset, k=3)
# Step 2: Extract reasoning flows from similar problems
reference_flows = [seed['reasoning_flow'] for seed in similar_seeds]
reference_budgets = [seed['thought_budget'] for seed in similar_seeds]
# Step 3: Prompt GPT-4o to generate reasoning for new problem
synthesis_prompt = f"""
Problem: {problem}
Similar problem reasoning flows (for guidance only):
{chr(10).join(reference_flows)}
Recommended thought budget (tokens): {max(reference_budgets)}
Generate step-by-step reasoning for this problem.
Maintain similar structure to reference flows.
Stay within thought budget.
"""
gpt4o_reasoning = call_gpt4o(synthesis_prompt, max_tokens=max(reference_budgets))
# Step 4: Extract solution from reasoning
solution = extract_solution(gpt4o_reasoning)
# Step 5: Filter by correctnessif verify_solution(solution, problem):
full_dataset.append({
'id': f'gen_{problem_id}',
'problem': problem,
'reasoning': gpt4o_reasoning,
'solution': solution,
'reasoning_flow': extract_reasoning_flow_simple(gpt4o_reasoning),
'thought_budget': len(gpt4o_reasoning.split()),
'source': 'gpt4o_guided'
})
return full_dataset
Quality Filtering: Remove low-quality or incorrect solutions
deffilter_generated_dataset(full_dataset, correctness_threshold=0.76):
"""
Filter dataset to keep only high-quality, correct solutions.
Target 76% accuracy after filtering (validated on validation set).
"""
filtered = []
for instance in full_dataset:
# Verify solution correctness
is_correct = verify_solution(instance['solution'], instance['problem'])
# Check reasoning quality metrics
reasoning_quality = evaluate_reasoning_quality(instance['reasoning'])
# Accept if correct and quality passes thresholdif is_correct:
filtered.append(instance)
elif reasoning_quality > 0.8and reasoning_quality > correctness_threshold:
# Keep near-correct with high quality reasoning (borderline cases)
filtered.append(instance)
# Calculate accuracy statistics
accuracy = sum(1for x in filtered if verify_solution(x['solution'], x['problem'])) / len(filtered)
print(f"Filtered dataset size: {len(filtered)}")
print(f"Accuracy: {accuracy:.1%}")
return filtered
defevaluate_reasoning_quality(reasoning):
"""
Heuristic evaluation of reasoning quality.
Checks for logical flow, completeness, clarity.
"""
quality_score = 0.0# Length heuristic: too short or too long is worse
token_count = len(reasoning.split())
if100 < token_count < 1000:
quality_score += 0.3# Logical markers: presence of reasoning keywords
reasoning_keywords = ['therefore', 'because', 'thus', 'step', 'first', 'next', 'finally']
keyword_count = sum(1for kw in reasoning_keywords if kw in reasoning.lower())
quality_score += min(0.3, keyword_count * 0.05)
# Math notation: equations and symbols in math problemsif'$'in reasoning or'='in reasoning:
quality_score += 0.2# No placeholder tokensif'<unk>'notin reasoning and'[...]'notin reasoning:
quality_score += 0.2return quality_score
Thought Budget Control: Manage reasoning token allocation
defapply_thought_budget_control(problem, reasoning_type='balanced'):
"""
Control maximum tokens for reasoning based on problem characteristics.
Prevents overthinking and excessive token generation.
"""
base_budget = {
'simple': 200, # For straightforward problems'balanced': 500, # For typical problems'complex': 1000, # For challenging problems'competition': 2000# For olympiad-level problems
}
if reasoning_type notin base_budget:
reasoning_type = 'balanced'
max_tokens = base_budget[reasoning_type]
# Adjust by problem difficulty heuristicif'difficult'in problem.lower() or'olympiad'in problem.lower():
max_tokens = min(2000, max_tokens * 2)
return max_tokens
defgenerate_with_budget(problem, model, thought_budget):
"""Generate reasoning respecting thought budget."""
reasoning = model.generate(
problem,
max_tokens=thought_budget,
temperature=0.7,
top_p=0.95
)
return reasoning
RL Fine-tuning Initialization: Use dataset for GRPO warm-start
definit_rl_with_long_cot_dataset(base_model, dataset, num_steps=100):
"""
Pre-train model on Long CoT Collection before RL fine-tuning.
Provides strong warm-start for GRPO optimization.
"""# Supervised fine-tuning on full dataset
optimizer = torch.optim.AdamW(base_model.parameters(), lr=1e-5)
for step inrange(num_steps):
batch = sample_batch(dataset, batch_size=32)
for example in batch:
# Forward pass
output = base_model(example['problem'], max_length=2048)
# Supervised loss
loss = torch.nn.functional.cross_entropy(
output.logits.view(-1, output.logits.shape[-1]),
example['reasoning_tokens'].view(-1)
)
# Backward
loss.backward()
optimizer.step()
optimizer.zero_grad()
return base_model
defcompare_rl_with_without_warmstart(base_model_no_warmstart, model_with_warmstart,
rl_steps=1000):
"""
Compare RL performance: cold-start vs. Long CoT warm-start.
Expect 2-3× performance gain with initialization.
"""
results = {}
# RL without warm-start (cold-start)
cold_start_model = train_with_rl(base_model_no_warmstart, steps=rl_steps)
results['cold_start'] = eval_model(cold_start_model)
# RL with Long CoT warm-start
warm_start_model = train_with_rl(model_with_warmstart, steps=rl_steps)
results['warm_start'] = eval_model(warm_start_model)
# Calculate improvement
improvement = results['warm_start']['accuracy'] / results['cold_start']['accuracy']
print(f"Warm-start improvement: {improvement:.2f}×")
return results
Practical Guidance
When to Apply:
Developing open-source reasoning models without access to o1
Limited budget for data annotation
Need strong baseline performance before RL fine-tuning
Want to reduce cold-start problem in reasoning model training
Dataset Acquisition:
Collect 1K seed samples manually from o1 (or use provided magpie-reasoning seed)
Extract reasoning flows and thought budgets from seeds
Use provided scripts to synthesize 100K samples via GPT-4o
Filter for correctness (expect ~76% accuracy after filtering)
Training with Long CoT Collection:
Pre-train 50-100 steps on full dataset (SFT phase)
Then apply RL fine-tuning (GRPO) for 1-2K steps
Expect 2-3× performance gains vs. cold-start RL alone
Performance Expectations:
Without warm-start (RL only): 22.7% on GPQA Diamond baseline
With Long CoT warm-start: 36.4%+ on GPQA Diamond
Generalization: Works on MATH, AIME, MMLU-Pro, multimodal tasks
RL convergence: Faster (2-3× improvement per RL step)
Thought Budget Guidelines:
Simple problems: 100-200 tokens
Standard problems: 300-600 tokens
Complex problems: 800-1500 tokens
Olympiad-level: 1500-2500 tokens
Configuration for Different Model Sizes:
7B models: 100K dataset sufficient, 50 SFT steps
14B models: 100K dataset, 100 SFT steps
70B+ models: Consider augmenting to 200K samples, 200 SFT steps
Common Issues:
Low correctness after filtering: Check synthesis prompts, may need better reference selection
Overthinking (excessive tokens): Reduce thought budget, apply earlier filtering
Poor RL convergence: Ensure SFT warm-start covers diverse problem types
Domain mismatch: Ensure seed dataset represents target domain well
Reference
100K Long CoT Collection available for community use. Base implementation on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct. Evaluated on MATH-500, AIME24, GPQA Diamond, MMLU-Pro benchmarks with RL training on 4-16 A100 GPUs using GRPO algorithm. Demonstrates strong generalization beyond seed domains and 2-3× RL performance improvement.
