golden-goose-task-synthesis

name	golden-goose-task-synthesis
title	Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.22975
keywords	["Task Synthesis","RLVR","Data Generation","Multiple-Choice","Training Data"]
description	Synthesize unlimited verifiable training tasks from unverifiable text by converting reasoning passages into multiple-choice problems. Creates higher-quality training data for RLHF systems without requiring new human labels.

Golden Goose: Unlimited Task Synthesis from Text

Problem

Models trained with Reinforcement Learning from Verifiable Rewards (RLVR) saturate quickly on limited, manually-created datasets. Performance plateaus prevent continued improvement on reasoning tasks like mathematics and coding, where verifiable correct/incorrect answers are expensive to generate at scale.

The core challenge is that RLVR requires ground-truth labels for verification—yet most internet text (textbooks, forum posts, research papers) cannot be automatically verified. Traditional RLHF approaches require expensive human feedback, creating a scaling bottleneck.

Core Concept

Golden Goose converts unverifiable reasoning text into verifiable multiple-choice training problems through fill-in-the-middle masking. The key insight is that intermediate reasoning steps naturally form correct answers, while other plausible continuations become high-quality distractors.

The process extracts reasoning-rich passages, masks crucial internal steps, generates diverse alternatives, and filters problems by difficulty. This transforms passive text into active verification tasks without human intervention.

Architecture Overview

• Passage Identification: Scan corpora (textbooks, AoPS forums, code repositories) for reasoning-rich content with clear intermediate steps
• Masking Strategy: Identify contiguous spans of crucial reasoning and replace with [MASK] token, preserving the underlying problem structure
• Choice Generation: Retain masked content as ground-truth answer; generate 8 diverse distractors using language model sampling
• Difficulty Filtering: Score by self-consistency; remove problems scoring <0.2 or >0.8 confidence to focus on medium-difficulty problems
• Scale: Results in 700K+ verifiable problems suitable for standard RL training

Implementation

Step 1: Extract Reasoning Passages

Identify passages containing multi-step reasoning by scanning educational texts for sufficient length and step diversity.

import re

def extract_reasoning_passages(documents, min_steps=3, min_length=500):
    """Extract passages likely to contain multiple reasoning steps."""
    passages = []
    for doc in documents:
        # Look for patterns with numbered steps, clear logic flow
         (doc) > min_length  doc.count() >= min_steps:
            passages.append(doc)
     passages