building-test-corpora

name	building-test-corpora
description	Building document corpora with controlled complexity levels (clean, mixed, messy) for RAG experiments. Use when creating test datasets, applying document degradation, or organizing documents for controlled experiments. Triggers on "build corpus", "create documents", "degrade documents", "test data".

Building Test Corpora

Create controlled document corpora with varying quality levels.

When to Use

• User says "build corpus" or "create test data"
• Need clean/mixed/messy document sets
• Applying controlled degradation to documents

Progress Checklist

[ ] 1. Identify source documents
[ ] 2. Create output directory
[ ] 3. For clean corpus: copy without modification
[ ] 4. For mixed corpus: add distractors + light degradation
[ ] 5. For messy corpus: apply medium/heavy degradation
[ ] 6. Validate corpus (file count, token estimate)
[ ] 7. Report statistics

Corpus Complexity Levels

Level	Characteristics	Degradation
clean	Official docs, formatted markdown	None
mixed	Clean + distractors	30% light
messy	Broken formatting, OCR artifacts	50% medium, 50% heavy

Core Rules

YOU MUST:

• Create output directories before writing files
• Apply degradation deterministically (set random seed)
• Validate corpus after creation

YOU MUST NOT:

• Modify source documents in place
• Mix degradation levels within a single corpus

Quick Implementation

def degrade_document(content: str, level: str) -> str:
    if level == "light":
        # Remove some headers, fix spacing
        content = re.sub(r"^#{1,3}\s+", "", content, flags=re.MULTILINE)
    elif level == "medium":
        # OCR-style errors + broken formatting
        content = content.replace("l", "1", 1) if random.random() < 0.3 else content
        content = re.sub(r"```\w+\n", "", content, count=2)
    elif level == "heavy":
        content = degrade_document(content, "medium")
        content = re.sub(r"\n\n", "\n", content, count=5)
    return content

Validation

After building, run:

python scripts/validate_corpus.py data/clean data/mixed data/messy

Expected output:

clean: 50 files, ~45K tokens
mixed: 70 files, ~65K tokens  
messy: 70 files, ~60K tokens
All corpora valid.

References

• See references/degradation_patterns.md for degradation details
• See references/distractor_creation.md for creating distractors
• See scripts/validate_corpus.py for validation script