| name | lcb-doctest-driven-development |
| description | Doctest-driven code generation for competitive-programming problems. The orchestrator first designs a quality gate of doctests (happy-path AND self-devised edge cases), verifies their expected values with a reference solution, injects them into the code generation prompt, then runs a reflection recovery loop until the code passes the gate. Use for any single-function / single-program coding task where correctness on unseen edge cases matters. The code-generating model never sees the hidden grading tests.
|
| version | 1 |
Doctest-Driven Development Skill
This skill turns a plain code generator into a test-first one. Instead of asking the
model to solve a problem blind, we build a quality gate of doctests up front — including
edge cases the public samples don't cover — and drive generation and repair against it.
A thin deterministic harness executes the phases below; each phase is a small tool in
tools/, loaded and run only when that phase is reached (progressive disclosure). The
model does all the reasoning (doctests, reference, code, repair); the harness only calls the
model and runs code.
The five patterns (human-native skill patterns)
- Progressive disclosure — this file stays short; each phase's detail lives in its own
tool module and is invoked only when needed. Deeper rationale:
references/patterns.md.
- Evidence ledger — every generated doctest carries a
confidence (HIGH/MED/LOW) and a
one-line justification. Only doctests backed by evidence become hard oracles.
- Quality gate — the code is judged by real execution against the doctests, not by vibes.
- Recovery loop — on failure, the model diagnoses the root cause and rewrites, bounded,
never-regressing.
- Doctest generation — the gate is authored by the model from the problem statement,
covering happy paths and self-devised edge cases.
Phases (run in order)
| Phase | Tool (load when reached) | What it does | Model call |
|---|
| 1. Classify | (harness) | TYPE A (stdin, empty starter_code) vs TYPE B (functional) | no |
| 2. Doctest generation + evidence ledger | tools/doctest_generator.py | model writes happy + edge doctests, each with a confidence score | yes |
| 3. Validate doctests | tools/validate_doctests.py | model writes a brute-force reference, harness repairs it until it passes the public samples, then recomputes each edge's expected value by running the reference (drops untrusted). Yields the trusted oracle = public samples + verified edges. Fallback: public-only. | yes |
| 4. Code generation | tools/code_generator.py | trusted doctests are injected into the prompt; model writes code to satisfy them | yes |
| 5. Quality gate | tools/quality_gate.py | execute the code against the oracle; report pass count + failing cases | no |
| 6. Recovery loop | tools/recovery_loop.py | while failing: show all failing cases, model diagnoses + rewrites; ≤5 attempts, never-regress | yes |
Shared helpers: tools/llm_client.py (model-agnostic LiteLLM proxy client) and
tools/execute.py (sandboxed stdin/functional execution + output comparison).
Contamination rule (critical)
The model is given only question_content, starter_code, and the public samples.
The reference is written from the statement; edge expected-values come from executing that
reference. Hidden/private tests are never shown to any model call — they are read only by
the separate grader (scripts/grade.py) after generation is complete.
How it is run
- Skill pipeline:
lcb_doctest/run_skill.py (orchestrates Phases 1–6).
- Baseline (no skill) pipeline:
lcb_doctest/run_baseline.py (plain generation).
- Grading (separate process, sees hidden tests):
scripts/grade.py.
