Use this skill whenever you are given a Python code file with missing code and asked to
complete it — whether that is a single incomplete line, an API invocation, or an entire
function body. This skill completes the missing code by first generating doctests as a
validation oracle, then generating the code, then iteratively fixing the code until it
passes its own doctests before delivering the result.
Trigger for:
- Any prompt containing a Python code file where code is missing or truncated
- Requests like "complete this function", "fill in the missing code", "implement this"
- JSON records with a "prompt" field containing Python code and a "metadata" field with
either "function_name" (function body task) or "line_no" (line completion task)
- Any RepoEval-style code completion benchmark instance
- Situations where generated code must be validated before delivery
Do not trigger for:
- General code questions or explanations with no missing code to fill in
- Requests to write entirely new files from scratch w
Use this skill whenever you are given a Python code file with missing code and asked to
complete it — whether that is a single incomplete line, an API invocation, or an entire
function body. This skill completes the missing code by first generating doctests as a
validation oracle, then generating the code, then iteratively fixing the code until it
passes its own doctests before delivering the result.
Trigger for:
- Any prompt containing a Python code file where code is missing or truncated
- Requests like "complete this function", "fill in the missing code", "implement this"
- JSON records with a "prompt" field containing Python code and a "metadata" field with
either "function_name" (function body task) or "line_no" (line completion task)
- Any RepoEval-style code completion benchmark instance
- Situations where generated code must be validated before delivery
Do not trigger for:
- General code questions or explanations with no missing code to fill in
- Requests to write entirely new files from scratch with no existing context
- Debugging tasks where all code is already present
- Doctest generation for already-complete functions (different skill)
compatibility
{"requires":"bash_tool, Python 3.7+","optional":"str_replace (for in-place file editing)","note":"If bash_tool is unavailable, the recovery loop cannot execute doctests. In that case, generate doctests and code but flag that validation was skipped."}
Code Completion with Doctest-Driven Validation
This skill completes missing Python code — either a full function body or the remainder
of a truncated line — using a test-first approach: generate doctests that define the
contract, generate code that fulfils it, then iteratively fix the code until it passes.
The order matters. Doctests before code forces you to commit to what the function should
do before deciding how it does it. Misunderstandings caught at the doctest stage cost
one iteration. Misunderstandings caught after five rounds of code generation cost
everything. The recovery loop then ensures you never deliver code that fails its own tests.
This is the NO-PROGRESSIVE-DISCLOSURE ablation. In the full skill, the doctest
generation rules, the recovery-loop logic, and the injection/delivery spec live in
separate reference files that are opened ONLY when a step needs them. Here, all of
that content is inlined into this one file and is resident from the first token.
Nothing is loaded on demand. Read the whole thing before processing instances.
Everything you need is in THIS file:
the 11-step workflow (below),
the complete Doctest Generation Guide (Appendix A),
the complete Recovery Loop Reference (Appendix B),
the complete Injection / Delivery Reference (Appendix C).
The only external artifact is the runner script run_doctest.py, which is executed,
not read into context.
Understanding your inputs
You will receive a JSON record in one of two shapes. The single metadata key that differs
between them is how you tell the tasks apart — check it explicitly, never infer from the
prompt shape.
Function body completion — the prompt ends right after a function's docstring closing
""" (or after the signature : if there is no docstring). The body is entirely absent.
No pass, no ..., nothing. Generate the complete indented body.
{
"prompt": "...all code up to and including the closing \"\"\" of the docstring...",
"metadata": {
"task_id": "CarperAI--trlx/idx",
"ground_truth": " the complete function body\n ...",
"fpath_tuple": ["CarperAI_trlx", "trlx", "pipeline", "__init__.py"],
"context_start_lineno": 0,
"lineno": 19,
"function_name": "register_datapipeline"
}
}
Line completion — the prompt ends mid-line. Generate only the characters that
complete that one line. Nothing more — no newline, no next line.
{
"prompt": "...all code up to the truncation point, ending mid-line...",
"metadata": {
"task_id": "huggingface_diffusers/0",
"ground_truth": " StableDiffusionInpaintPipelineLegacy,",
"fpath_tuple": ["huggingface_diffusers", "tests", "pipelines", "test_stable_diffusion_inpaint_legacy.py"],
"context_start_lineno": 0,
"line_no": 28
}
}
Routing rule:"function_name" in metadata → FUNCTION_BODY task.
"line_no" in metadata → LINE task. These are mutually exclusive.
Configuration
Parameter
Default
What it controls
MAX_RECOVERY_ITERATIONS
5
Fix attempts before escalating. Range: 1–7
DOCTEST_SCOPE
"targeted"
"targeted" = target function only; "file" = also generate for functions the target calls internally
INJECT_DOCTESTS
true
Whether passing doctests stay in the final output permanently
BEST_ATTEMPT_ON_FAILURE
true
On exhausted retries, deliver best attempt rather than nothing
The workflow
Steps run in strict order. Each produces something the next depends on.
Step 1 → Analyse the file
Step 2 → Route to task type
Step 3 → Load additional context (only what is needed)
Step 4 → Classify the target function
Step 5 → Pre-generation quality gate
Step 6 → Generate doctests [see Appendix A, below]
Step 7 → Validate doctest structure
Step 8 → Generate first candidate code
Step 9 → Recovery loop [see Appendix B, below]
Step 10 → Post-loop quality gate
Step 11 → Inject and deliver [see Appendix C, below]
Step 1 — Analyse the file
Read the entire prompt field before doing anything else. Build a mental model of the
codebase so that whatever you generate fits naturally into it.
As you read, note:
Every import statement and what it provides
Module-level variables and constants
Class names, their methods, and inheritance relationships
Other function names visible in the prompt and what they do
The naming style in use (snake_case, camelCase — infer from existing code)
How existing functions handle errors: raise, return None, return a sentinel value?
What types existing functions return
Any decorators on the target function
The exact docstring of the target function, verbatim, if present
Stop here if the prompt is empty or unparseable. Report the error rather than proceeding
with no context. The rest of the workflow depends entirely on what you learn here.
Step 2 — Route to task type
Check the metadata keys:
"function_name" present → FUNCTION_BODY task. Record the function name and
metadata["lineno"] as the line where the body starts.
"line_no" present → LINE task. Record metadata["line_no"] as the line
being completed.
Neither present → stop. Report that the task type cannot be determined.
For FUNCTION_BODY: generate everything from the first indented line of the body
through the final return or expression. Match the file's indentation (4 spaces standard).
Do not alter the signature or the docstring.
For LINE: generate only the characters that come after the last character of the
prompt on that same line. No leading newline. No additional lines. Just the remainder of
the syntactic construct that was cut off.
Step 3 — Load additional context
The prompt is your primary source. Before loading anything else, exhaust what it already
tells you. Only go further if the target function references names that do not appear
anywhere in the prompt and cannot be inferred from the import statements.
When you do need more context, use fpath_tuple to identify which files in the same
directory or package would define the missing names. Load the minimum needed — one or
two files at most.
Never load the repository's test files. Those are the external ground truth that runs
after delivery. Loading them contaminates the completion.
Every additional file you load costs context tokens and adds noise. The best completions
come from using what is already in front of you well.
Step 4 — Classify the target function
Before generating any doctests, classify the target function into the tier that best
describes what it does. The tier determines what kind of doctests are possible and what
the recovery loop is actually testing.
Tier
Type
Signal
Doctest strategy
1
Pure / Deterministic
No external calls, same output every time
Full input → output examples
2
Contract-Testable
Non-deterministic output but testable type/shape/range
Test isinstance, len, bounds
3
Setup-Assisted
Needs temp object, class instantiation, or asyncio.run
Set up context inline in doctest
4
Error-Path Only
Calls DB, API, network, filesystem
Test only input validation and error raises
5
Untestable
Even error paths need live external state
Document with TODO, skip doctest
Special case for LINE tasks: First determine what syntactic context the truncated
line sits in:
Inside a function body → classify that function using the tier table above
Inside an import block (from x import () → Tier 5, no doctest possible
Module-level assignment or constant → Tier 5, no doctest possible
Class attribute declaration → Tier 5, no doctest possible
For Tier 5 contexts in LINE tasks, skip doctest generation entirely and proceed directly
to Step 8. Record that validation was skipped and why.
See Appendix A (inlined below) for detailed guidance on each tier's generation
patterns before proceeding to Step 6.
Step 5 — Pre-generation quality gate
Before writing any doctest or any code, confirm you can answer these questions. If you
cannot answer one, make the assumption explicit and record your confidence level (high /
medium / low). A wrong assumption here propagates through every subsequent step.
What does this function do? (source: docstring, function name, call sites in the prompt)
What input types does it accept?
What does it return, and in what type?
What exceptions should it raise, and under what conditions?
Are all imports this implementation will need already present in the prompt?
For LINE tasks: what syntactic construct is the truncated line completing?
If a critical question has no reasonable answer even after making explicit assumptions,
pause and ask rather than guessing silently. Guessing silently on a wrong foundation
means the recovery loop spends all its iterations fixing the wrong thing.
Step 6 — Generate doctests
The complete doctest generation rules are inlined below in Appendix A. Apply them directly.
Generate doctests appropriate for the tier you identified in Step 4. The goal is to
define the function's contract before implementing it — what goes in, what comes out,
what gets raised. If you find yourself unable to write a meaningful expected output,
that is a signal you have misclassified the tier, not a reason to write a vague test.
For Tier 1–2 FUNCTION_BODY tasks: generate 2–6 doctests covering:
The typical happy path with a concrete input and concrete expected output
At least one edge case (empty input, zero, None, boundary value)
Any exception the function is documented to raise
For Tier 3 FUNCTION_BODY tasks: generate setup inline in the doctest block before
the assertion. Use asyncio.run() for async functions. Instantiate required objects
directly in the doctest.
For Tier 4 FUNCTION_BODY tasks: generate doctests only for input validation and
error-raise paths. Do not attempt to test the external call itself.
For LINE tasks inside a function (Tier 1–3): generate a doctest for the containing
function that exercises the line being completed as part of its normal execution.
All expected output values must be concrete and deterministic. No random output, no
memory addresses, no timestamps.
Step 7 — Validate doctest structure
Before running anything, verify the doctests are syntactically correct. A malformed
doctest will either silently pass everything (masking bugs) or always fail (burning
recovery iterations on a formatting problem rather than a code problem).
Check every doctest block for:
>>> prefix with exactly one space after the arrows
... prefix on continuation lines with exactly one space
Expected output on the very next line after >>> — no blank line between them
<BLANKLINE> used wherever expected output contains a blank line
Exception format is exactly: Traceback (most recent call last): then ... then
ExceptionType: message
No ... as a wildcard in expected output unless # doctest: +ELLIPSIS is on the
same >>> line
Fix any formatting problems before proceeding. Do not carry malformed doctests into the
recovery loop — the loop assumes the tests are well-formed and diagnoses code failures,
not test failures.
Step 8 — Generate first candidate code
Now generate the completion using everything gathered in Steps 1–7:
For FUNCTION_BODY tasks:
Generate the full body, properly indented from first line to last
Stay consistent with the imports already in the file — do not introduce new ones
unless absolutely necessary and not already importable from what is present
Honour the naming conventions and error-handling patterns observed in Step 1
The body must be consistent with the expected outputs you committed to in Step 6
Do not alter the signature or the docstring
For LINE tasks:
Generate only the remainder of the truncated line
Start at the exact character position where the prompt ends
No leading whitespace, no leading newline
End at the natural conclusion of the syntactic construct (closing bracket, quote,
comma, or nothing if the line ends bare)
Do not generate the next line
Step 9 — Recovery loop
The complete recovery-loop logic is inlined below in Appendix B. Apply it directly.
Write the candidate code to a temporary file alongside the generated doctests, then run:
The runner returns structured output showing exactly which doctests passed, which failed,
what was expected, and what was actually produced. Use this output to make targeted fixes
to the code — not to the doctests. The doctests represent the contract. The code is what
changes.
Key rules for the loop:
Track the best attempt across iterations — the candidate with the most passing doctests.
If iterations exhaust before all doctests pass, the best attempt is what gets delivered,
not the last attempt.
If the same failure repeats after two consecutive fix attempts, stop trying the same
approach. Either the function has been misclassified (revisit Step 4 and reclassify) or
the doctest expects something the codebase cannot support (revisit Step 6 and adjust
the expected value based on what you now know). Do not burn remaining iterations
on an approach that has already failed twice.
On exhaustion, inject only the doctests that passed in the best attempt. Replace each
failing doctest with a structured TODO comment — never inject a known-failing doctest
into the output file. A broken example in a codebase is worse than no example.
Failing doctest replacement format:
# TODO: doctest-completion could not produce a passing example for this case.
# Attempted: MAX_RECOVERY_ITERATIONS iterations
# Last failure:
# Input: <what was tested>
# Expected: <what the doctest expected>
# Got: <what the code actually produced>
# To fix: <your diagnosis of what would need to change>
Step 10 — Post-loop quality gate
Before delivering anything, verify:
Does the completion cover only what was missing — no changes to the existing prompt?
Is indentation consistent throughout (no mixed tabs/spaces, correct depth)?
For LINE tasks: does the completion attach cleanly to the last character of the prompt
with no spurious leading space or newline?
Are there syntax errors in the completion (unclosed brackets, mismatched quotes,
invalid Python)?
Does the completion reference any names not in scope (not in imports, not in globals,
not in the function's own parameters)? Flag these but do not block on them — they may
be inherited attributes or injected by a decorator.
If retries exhausted: is the TODO replacement ready for each failing doctest?
Block on syntax errors. Flag but proceed on scope warnings.
Step 11 — Inject and deliver
The complete injection/delivery spec is inlined below in Appendix C. Apply it directly.
Assemble the final output:
The original prompt, unchanged
The completion appended at the correct position
If INJECT_DOCTESTS = true: passing doctests embedded in the function's docstring,
after the original docstring text and before the closing """
Failing doctests replaced with TODO comments as specified in Step 9
Deliver in this structure:
task_id: <from metadata>
task_type: FUNCTION_BODY or LINE
status: SUCCESS (all doctests pass) or PARTIAL (best attempt, some failing)
iterations_used: <n>
doctests_passing: <n of passing> / <n total generated>
completion: <the generated code only — not the full file>
full_output: <complete prompt + completion with doctests embedded>
escalation: null or structured report if PARTIAL
Then tell the user, in plain language:
What tier the function was classified as and why that mattered
How many iterations the recovery loop needed
Which doctests passed and what they verify
For PARTIAL status: what specifically failed, what the diagnosis is, and what a
developer would need to do to resolve it
Be specific. "Skipped DB call testing — validated all 3 input validation paths instead"
is useful. "Some tests were skipped" is not.
What to tell the user: "Tier 1 function — pure and deterministic, so full input/output
doctests were possible. All 3 cases passed on the first attempt: zero input, typical input,
and the negative input error path."
What this skill does not do
Run the developer-written ground truth tests — those execute externally after delivery
Modify the function signature or existing docstring prose
Generate completions for multiple functions in one pass
Load repository test files (doing so contaminates the completion)
Fix bugs in already-complete code (different task entirely)
APPENDIX A — Doctest Generation Guide (inlined)
This is Appendix A. It is part of this single monolithic skill file (not loaded separately). Its job is to tell you exactly
how to generate doctests that are syntactically correct, semantically meaningful, and
appropriate for the function you are completing — based on the tier you identified in
Step 4.
The doctests you generate here are not the deliverable. They are the validation oracle
that the recovery loop uses to judge whether the code you generate is correct. Get them
right here and the recovery loop can do its job. Get them wrong and the loop either
tests the wrong thing or burns all its iterations on a formatting problem rather than
a code problem.
Doctest syntax — the exact rules
These rules come directly from Python's doctest specification. Every generated doctest
must follow all of them, no exceptions.
>>> — exactly three > characters followed by exactly one space. Nothing else before
the code on that line.
... — exactly three . characters followed by exactly one space. Used for any line
that continues a multi-line statement (function definitions, if blocks, loops, with
blocks, multi-line calls).
Expected output placement
The expected output must appear on the very next line after the final >>> or ... line
of a statement. No blank line between the statement and the expected output.
# CORRECT
>>> factorial(5)
120
# WRONG — blank line between statement and expected output
>>> factorial(5)
120
Blank lines inside expected output
If the function's actual output contains a blank line, you cannot use a real blank line
in the doctest — a blank line signals the end of expected output. Use <BLANKLINE>
instead.
>>> print_with_gap()
first line
<BLANKLINE>
second line
Exception format
Exceptions have a strict three-part structure:
>>> function_that_raises(bad_input)
Traceback (most recent call last):
...
ExceptionType: message text here
The first line must be exactly Traceback (most recent call last): — no variation.
The middle is exactly ... (four spaces then three dots). This is not the ELLIPSIS
option — it is always literal in exception doctests and always works regardless of the
ELLIPSIS flag.
The final line is the exception type and message. This is what doctest actually checks.
The traceback body is always ignored.
The exception message must match exactly unless you add # doctest: +ELLIPSIS and use
... as a wildcard within the message.
Directives
Place directives as comments on the same >>> line they apply to:
>>> some_function() # doctest: +ELLIPSIS
<SomeObject at 0x...>
>>> another_function() # doctest: +NORMALIZE_WHITESPACE
a b c
>>> expensive_example() # doctest: +SKIP
Available directives relevant to code completion:
Directive
When to use
+ELLIPSIS
Output contains memory addresses, UUIDs, or other unpredictable substrings
+NORMALIZE_WHITESPACE
Output whitespace may vary (e.g., repr of dicts, formatted strings)
+SKIP
Example must be shown for documentation but cannot run in isolation
Do not use +SKIP as an escape hatch for laziness. Only use it when the example
genuinely cannot run — for example, a function that requires a live server connection
where even a stub test is impossible.
What NOT to test directly
Avoid testing these in expected output — they will produce non-deterministic results
that fail on different machines or across runs:
Memory addresses: <MyObject at 0x7f3a...> — use # doctest: +ELLIPSIS
Dictionary order in Python < 3.7 (not a concern for 3.7+, but be aware)
Floating point with many decimal places — use round() in the doctest input
Timestamps, UUIDs, random values — test the type or shape instead
Any repr() that includes internal state subject to change
Tier-by-tier generation guide
Tier 1 — Pure / Deterministic
The function takes inputs, computes something, returns a result. Same input always gives
the same output. No external calls, no state, no I/O.
Generate:
2–4 concrete input → output examples
At least one edge case: zero, empty string, empty list, None where accepted, boundary values
All documented exception paths
Example — string utility:
def truncate(text, max_length, suffix="..."):
"""Truncate text to max_length, appending suffix if truncated.
>>> truncate("hello world", 8)
'hello...'
>>> truncate("hi", 10)
'hi'
>>> truncate("hello world", 8, suffix="—")
'hello w—'
>>> truncate("", 5)
''
>>> truncate("hello", 0)
Traceback (most recent call last):
...
ValueError: max_length must be positive
"""
The function's output is non-deterministic (order varies, contains IDs, timing-dependent)
but its shape, type, length, or range is predictable.
Generate:
Tests that check isinstance(result, expected_type)
Tests that check len(result) or result in valid_range
Tests that check specific keys exist in a returned dict
Tests that check invariants that must always hold
Exception paths remain fully testable
Example — function returning a set or shuffled list:
def unique_words(text):
"""Return the set of unique words in text.
>>> result = unique_words("the cat sat on the mat")
>>> isinstance(result, set)
True
>>> len(result)
5
>>> "cat" in result
True
>>> unique_words("")
set()
"""
Example — function returning a dict with known keys:
def parse_config(config_str):
"""Parse a KEY=VALUE config string into a dict.
>>> result = parse_config("host=localhost port=8080")
>>> isinstance(result, dict)
True
>>> result["host"]
'localhost'
>>> result["port"]
'8080'
>>> parse_config("")
{}
"""
Tier 3 — Setup-Assisted
The function needs context to run: a class instance, a temporary file, an async event
loop, or some initial state. The setup must happen inside the doctest block itself.
Generate:
Inline setup before the function call
Use asyncio.run() for async functions — do not use await directly in doctests
Instantiate required objects directly
Clean up if the setup creates side effects (though doctest isolation usually handles this)
Example — method on a class:
class Counter:
def __init__(self):
self.value = 0
def increment(self, amount=1):
"""Increment the counter by amount.
>>> c = Counter()
>>> c.increment()
>>> c.value
1
>>> c.increment(5)
>>> c.value
6
>>> c.increment(-1)
Traceback (most recent call last):
...
ValueError: amount must be positive
"""
Example — async function:
async def fetch_cached(key, cache):
"""Fetch a value from cache, returning None if missing.
>>> import asyncio
>>> cache = {"x": 42}
>>> asyncio.run(fetch_cached("x", cache))
42
>>> asyncio.run(fetch_cached("missing", cache)) is None
True
"""
Example — function needing a temp file:
def count_lines(filepath):
"""Count the number of lines in a file.
>>> import tempfile, os
>>> with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
... _ = f.write("line1\\nline2\\nline3\\n")
... tmp = f.name
>>> count_lines(tmp)
3
>>> os.unlink(tmp)
"""
Tier 4 — Error-Path Only
The function's core behaviour requires external infrastructure: a database connection,
a network call, a third-party API, the filesystem at a specific path. Testing the
happy path would require mocking the infrastructure, which is outside the scope of
inline doctests.
Generate:
Only the input validation and error-raise paths
Do not attempt to test the actual external call
Be explicit in a comment that happy-path testing requires integration setup
Example — function that writes to a database:
def save_user(db_conn, user_id, name):
"""Save a user record to the database.
# Happy-path testing requires a live database connection.
# Doctest covers input validation only.
>>> save_user(None, 1, "Alice")
Traceback (most recent call last):
...
ValueError: db_conn cannot be None
>>> save_user("conn", None, "Alice")
Traceback (most recent call last):
...
ValueError: user_id cannot be None
>>> save_user("conn", 1, "")
Traceback (most recent call last):
...
ValueError: name cannot be empty
"""
Example — function that makes an HTTP request:
def get_user_profile(user_id, api_client):
"""Fetch a user profile from the API.
# Live API testing not possible in doctest.
# Covers input validation only.
>>> get_user_profile(None, object())
Traceback (most recent call last):
...
ValueError: user_id must be a positive integer
>>> get_user_profile(-1, object())
Traceback (most recent call last):
...
ValueError: user_id must be a positive integer
"""
Tier 5 — Untestable / Skip
The function cannot be meaningfully tested in a doctest — even the error paths require
live external state. Or the completion target is not a function at all (it is an import
line, a constant, a class attribute declaration).
Do not generate doctests. Instead, document why:
# DOCTEST SKIPPED — Tier 5
# Reason: <why testing is not possible>
# Context: <what the line/function does>
# To test: <what would be needed for real testing>
Tier 5 triggers for LINE tasks:
Truncated line is inside an import or from ... import ( block
Truncated line is a module-level constant: MAX_RETRIES =
Truncated line is a class attribute: _registry: Dict[str, Any] =
Truncated line is inside a decorator definition with no callable context
Truncated line is a type alias: UserID =
In all these cases: proceed directly to Step 8 (code generation). Record in the output
that validation was skipped and state the reason.
The doctest block placement rules
For FUNCTION_BODY tasks
Doctests go inside the function's docstring, after the prose description and before the
closing """. Follow this structure exactly:
def function_name(args):
"""Original one-line summary.
Any additional prose from the original docstring goes here,
preserved verbatim.
>>> function_name(typical_input)
expected_output
>>> function_name(edge_case)
expected_edge_output
>>> function_name(bad_input)
Traceback (most recent call last):
...
ExceptionType: message
"""
# body goes here
If the original docstring has no prose (it only has the """ close), add a blank line
before the first >>>:
Find the function that contains the truncated line and add doctests to that function's
docstring using the same rules as above. If that function has no docstring, create one.
Minimum and maximum doctest counts
Tier
Minimum
Maximum
Notes
1
2
6
Always include edge case + exception if documented
2
2
4
Contract checks count as separate tests
3
1
4
Setup complexity limits how many are practical
4
1
3
Error paths only — don't pad with redundant cases
5
0
0
Skip entirely
More is not always better. A doctest suite with 6 redundant happy-path cases and no
edge cases is weaker than one with 3 cases that actually cover the boundaries. Choose
cases that would catch the bugs most likely to appear in a naive implementation.
This is Appendix B. It is part of this single monolithic skill file (not loaded separately). It governs how the recovery loop
runs, how failures are diagnosed, how fixes are targeted, and what happens when
iterations are exhausted.
The recovery loop is the core mechanism that separates this skill from a one-shot code
generator. Its job is not to retry blindly — it is to read the exact failure output from
the runner, understand why the code failed, make a targeted fix that addresses that
specific failure without breaking what already passes, and repeat until all doctests pass
or the iteration budget is spent.
The loop structure
Before the first iteration, initialise these tracking variables:
current_iteration = 1
best_attempt = <the code from Step 8>
best_pass_count = 0
best_iteration = 1
final_code = None ← stays None until all doctests pass
Then run the loop:
WHILE current_iteration <= MAX_RECOVERY_ITERATIONS:
1. Write candidate to temp file
2. Run the test runner
3. Parse the output
4. Check results
→ IF all pass: set final_code, break
→ IF some pass: update best_attempt if improved, then diagnose and fix
→ IF none pass: diagnose and fix
5. Check for repeated failure (same failure twice → reclassify or adjust)
6. Increment current_iteration
AFTER LOOP:
→ IF final_code is set: proceed to Step 10 (success)
→ ELSE: proceed to Step 10 (partial — use best_attempt)
Step-by-step iteration procedure
1. Write the candidate to a temp file
Write the complete candidate — the original prompt plus the generated completion with
doctests embedded — to a temporary file at a consistent path:
/tmp/completion_candidate.py
Overwrite this file on each iteration. The runner uses PID-based module naming
internally so it never reads a cached version from a previous iteration.
The file must be a complete, runnable Python file. It must include:
All imports from the original prompt
All module-level globals and constants from the original prompt
The target function with its docstring (including the generated doctests)
The generated function body
Do not write only the function in isolation — imports and globals are often required
for the function to execute at all.
Capture the full stdout. The runner will never mix its output with doctest internals —
everything you receive is structured and machine-parseable.
3. Parse the output
Read the structured output. Key fields:
STATUS: PASSED | FAILED | ERROR | SKIPPED
TOTAL: <n> ← total doctests attempted
PASSED: <n> ← how many passed this iteration
FAILED: <n> ← how many failed
--- Failure N ---
TEST: <the >>> line that failed>
EXPECTED: <what the doctest expected>
GOT: <what the code produced>
LOCATION: <file>:<line>
---
SUMMARY: <human-readable summary>
STATUS: ERROR means the candidate file has a syntax error or import failure. The
code cannot run at all. Treat this as zero passes — diagnose as SYNTAX or IMPORT class
failure and fix before anything else.
STATUS: SKIPPED means no doctests were found. This should not happen during the
recovery loop (doctests were validated in Step 7). If it does, check that the temp file
was written correctly and contains the docstring.
4. Update best attempt tracking
IF PASSED count > best_pass_count:
best_attempt = current candidate code
best_pass_count = current PASSED count
best_iteration = current_iteration
Track the best attempt across all iterations, not just the most recent one. The most
recent iteration is not necessarily the best — a fix that resolves one failure can
accidentally regress another.
5. Check the exit condition
IF STATUS == PASSED (all doctests pass):
final_code = current candidate
BREAK — do not run more iterations
Exit as soon as all doctests pass. Do not run extra iterations "just to be sure."
Failure diagnosis
When the runner reports failures, diagnose each one before writing any new code. Making
a fix without a diagnosis produces random changes that as often regress passing tests as
fix failing ones.
Failure classification
Classify each failure into one of these categories. The category determines the fix
strategy.
Fix strategy: Read the SyntaxError line number. Fix that specific line. Do not rewrite
the entire function — the error is localised. Common sources: missing : after if/
for/def, unclosed ( or [, inconsistent indentation mixing tabs and spaces,
f-string with unescaped {.
IMPORT — The candidate imports something not available.
Cause: Generated code uses a module that is not imported at the top of the file, or
uses a name from an import that was not in the original prompt.
Fix strategy: Check the original prompt's import list carefully. If the needed import
is present under an alias (e.g., import numpy as np), use the alias. If the import
is genuinely missing from the prompt, either find a way to implement without it or add
the import — but only if it is a standard library module. Do not add third-party imports
that were not in the original prompt.
WRONG_VALUE — The code ran and returned something, but the value was wrong.
Cause: Logic error in the generated code. The implementation computes the wrong result
for this input.
Fix strategy: Read the specific TEST line and the EXPECTED vs GOT values. Reason
about what the code does with that specific input and where the logic diverges from the
expected output. Make the minimal change that corrects the computation for this input
without breaking the inputs that already pass.
Common sources: off-by-one errors, wrong operator (// vs /, and vs or), wrong
variable referenced, missing case in conditional, incorrect base case in recursion.
WRONG_TYPE — The code returned the right kind of thing but in the wrong type.
Runner signal: STATUS: FAILED, GOT: <something> where the type is visible (e.g.,
GOT: [1, 2, 3] but EXPECTED: (1, 2, 3))
Cause: Return type does not match what the doctest expects.
Fix strategy: Check the function's type hints and docstring for the intended return
type. Wrap or convert the return value. If the doctest uses a type-checking pattern
(isinstance(result, list)) and the type is wrong, fix the return type in the code.
MISSING_EXCEPTION — The code should raise an exception but does not.
Runner signal: STATUS: FAILED, EXPECTED: Traceback (most recent call last): ... ExceptionType: message, GOT: <some return value or nothing>
Cause: The input validation or guard clause for this error case is missing or
incorrectly placed.
Fix strategy: Add the missing guard. Check the original prompt's docstring — it likely
describes the conditions under which exceptions are raised. Add the check before the
main logic, not after.
WRONG_EXCEPTION — The code raises an exception, but the wrong one.
Cause: The generated code crashes on a valid input. Common causes: index out of range,
key error in dict lookup, attribute access on None, division by zero on an edge case
input, recursion without a correct base case.
Fix strategy: Read the exception type and message from GOT. Trace which line of the
generated code would produce that exception for the specific input shown in TEST.
Add a guard, fix the bounds check, or correct the logic that leads to the crash.
REPR_MISMATCH — The code produces the correct value but the string representation
differs.
Runner signal: STATUS: FAILED, EXPECTED: {'a': 1}, GOT: {'a': 1, 'b': 2} (extra
keys) or EXPECTED: [1, 2, 3], GOT: [1, 2, 3] (looks identical but differs in
whitespace or repr format)
Cause: The doctest expected output was written based on the wrong repr, or the
NORMALIZE_WHITESPACE flag did not cover the difference, or the output contains extra
items.
Fix strategy:
If extra items: the logic is producing more output than expected — find what should
be filtered.
If repr looks identical but fails: run the specific expression in isolation to see
the actual repr. There may be invisible characters or encoding differences.
If it is a floating point issue: use round() in the doctest input, or switch to a
type check.
The repeated failure rule
If the same failure appears in two consecutive iterations — same TEST line, same
GOT value, same failure category — your current fix strategy is not working. Do not
apply the same fix a third time.
Instead, do one of:
Option A — Reclassify the function (revisit Step 4)
The function may be higher-tier than you assumed. If you are testing a Tier 1 function
as if it were pure but it actually calls external state, the doctest will always fail
regardless of how correct the code is. Reclassify and adjust the doctest scope
accordingly.
Option B — Adjust the doctest expected value (revisit Step 7)
You may have written an expected output that the codebase cannot actually produce. For
example:
You expected 42 but the function can only produce 42.0 (float vs int)
You expected a sorted list but the function returns items in insertion order
You expected a specific exception message that the underlying library produces
differently
If the expected output in the doctest is wrong, fix the doctest — not the code. The
doctest must reflect what is actually achievable given the codebase. Do not change the
doctest to pass trivially (e.g., changing EXPECTED: 42 to EXPECTED: 99 to match
broken code). Only change it if the expected value was genuinely wrong based on what
you now know the function should produce.
Option C — Reduce the doctest scope
If the failing doctest tests something that cannot be made to pass within the constraints
of the codebase (e.g., a Tier 4 function being asked to produce a DB result), remove
that specific doctest case and replace it with a TODO comment. Do not remove passing
doctests. Only remove the specific failing case that cannot be resolved.
Fixing without regression
Every fix must satisfy two conditions:
It addresses the diagnosed failure
It does not regress any doctest that was passing in the previous iteration
Before writing the fixed code, list every passing doctest from the current iteration.
After writing the fix, mentally trace each passing doctest through the new code to
confirm it still produces the right output.
The most common regression cause: a fix that adds a special case for one input breaks
the general case for another. For example, adding if n == 0: return 1 to fix a
base-case failure while accidentally shadowing correct logic for n > 0.
Exhaustion handling
When current_iteration > MAX_RECOVERY_ITERATIONS and final_code is still None:
What gets injected
Use best_attempt — the candidate code from the iteration with the highest pass count.
If two iterations had equal pass counts, use the earlier one (lower iteration number is
the simpler fix and less likely to have introduced regressions).
Doctest handling on exhaustion
Passing doctests in best_attempt: inject them permanently into the docstring.
They are verified and correct.
Failing doctests in best_attempt: replace each one with a TODO comment block.
Never inject a known-failing doctest into the output file.
TODO comment format:
# TODO: code-completion — doctest could not be resolved after MAX_RECOVERY_ITERATIONS iterations
# Failed test: <the >>> line>
# Expected: <expected output>
# Got: <actual output>
# Failure class: <WRONG_VALUE | MISSING_EXCEPTION | UNEXPECTED_EXCEPTION | etc.>
# Diagnosis: <your best understanding of why this cannot be fixed automatically>
# To resolve: <what a developer would need to do to make this pass>
Escalation report
Generate this report and include it in the Step 10 output:
ESCALATION REPORT
─────────────────────────────────────────────────────────────
task_id: <from metadata>
task_type: FUNCTION_BODY | LINE
iterations_attempted: <MAX_RECOVERY_ITERATIONS>
best_iteration: <n>
doctests_total: <n>
doctests_passing: <n in best_attempt>
doctests_failing: <n in best_attempt>
Failing tests and diagnosis:
[1] TEST: <>>> line>
Expected: <value>
Got: <value>
Class: <failure class>
Diagnosis: <why it failed>
Fix needed: <what would resolve it>
[2] ...
Recommendation:
<One paragraph — what the developer should look at,
what assumption was likely wrong, and what approach
would make these tests pass.>
─────────────────────────────────────────────────────────────
Iteration budget guidance
Situation
Expected resolution
Simple logic error (wrong operator, missing case)
Iteration 1–2
Missing guard clause (exception not raised)
Iteration 1
Type mismatch (list vs tuple, int vs float)
Iteration 1
Complex multi-case logic
Iteration 2–3
Misclassified tier discovered mid-loop
Iteration 3–4 (after reclassification)
Structural misunderstanding of function contract
Iteration 4–5 then escalate
If you reach iteration 3 with zero passing doctests, stop and reclassify before
continuing. Continuing without reclassification when nothing is passing means
the contract assumption is fundamentally wrong, and more iterations will not help.
What the recovery loop never does
Never modifies the function signature
Never modifies the original docstring prose
Never removes a passing doctest to make the count look better
Never changes an expected value in a doctest to match broken code
Never introduces new imports that were not in the original prompt (except standard
library if genuinely required)
Never generates a completion for a different function than the target
Never stops early because "the code looks right" — only the runner's output
determines success
APPENDIX C — Injection / Delivery Reference (inlined)
This is Appendix C. It is part of this single monolithic skill file (not loaded separately). It governs how the validated
completion is assembled back into the original code context and delivered.
Injection is the final irreversible step. Everything upstream — context analysis,
doctest generation, the recovery loop — exists to ensure that what gets injected
here is correct. The rules in this file exist to ensure the injection itself
introduces no new problems: no indentation shifts, no docstring corruption, no
misplaced characters, no line boundary errors.
What injection means for each task type
FUNCTION_BODY tasks
The prompt field contains all code up to and including the function's docstring
closing """ (or the function signature : if there was no docstring). The body
is entirely absent. Injection appends the generated body immediately after the
prompt ends.
[prompt content]
def register_datapipeline(name):
"""Decorator used to register a CARP architecture.
Args:
name: Name of the architecture
>>> register_datapipeline('test')(dict) # doctest: +ELLIPSIS
<class 'dict'>
"""
← prompt ends here (after closing """)
def register_class(cls, name): ← injected body starts here
_DATAPIPELINE[name] = cls
...
return cls
The first line of the injected body begins on the line immediately after the
prompt's last character. A newline already exists at the end of the """ line —
do not add another one.
LINE tasks
The prompt field ends mid-line. The completion attaches directly to the last
character of the prompt with no separator of any kind.
[prompt content]
from diffusers import (
AutoencoderKL,
DDIMScheduler,
PNDMScheduler,
← prompt ends here, mid-line, after the comma
StableDiffusionInpaintPipelineLegacy, ← completion attaches here
For LINE tasks there is no doctest injection — line completions do not have a
function docstring to inject into. The output is simply the prompt concatenated
with the completion string.
Doctest injection — FUNCTION_BODY tasks only
When INJECT_DOCTESTS = true (the default), passing doctests are embedded
permanently in the function's docstring.
Placement within the docstring
Doctests go after all existing prose and before the closing """. A blank line
separates the prose from the first >>> line.
Original docstring (prose only):
def factorial(n):
"""Return n! for non-negative integer n.
Args:
n: A non-negative integer.
"""
After doctest injection:
def factorial(n):
"""Return n! for non-negative integer n.
Args:
n: A non-negative integer.
>>> factorial(0)
1
>>> factorial(5)
120
>>> factorial(-1)
Traceback (most recent call last):
...
ValueError: n must be non-negative
"""
If the original docstring is a single line (no newline after text):
Expand it to multi-line format first, then inject:
# Before
def add(a, b):
"""Return the sum of a and b."""
# After
def add(a, b):
"""Return the sum of a and b.
>>> add(2, 3)
5
>>> add(-1, 1)
0
"""
If the function had no docstring at all:
Create one containing only the doctests:
# Before
def add(a, b):
return a + b
# After (with injected docstring)
def add(a, b):
"""
>>> add(2, 3)
5
>>> add(-1, 1)
0
"""
return a + b
Handling failing doctests on exhaustion
When the recovery loop exhausted its iterations without full resolution, some
doctests failed. Inject only the passing ones. Replace each failing doctest with
a TODO comment block inside the docstring.
def complex_function(x, config):
"""Process x according to config.
>>> complex_function(0, {})
Traceback (most recent call last):
...
ValueError: config cannot be empty
# TODO: code-completion — doctest could not be resolved after 5 iterations
# Failed test: complex_function(42, {"mode": "fast"})
# Expected: {'result': 84, 'status': 'ok'}
# Got: {'result': 84, 'status': 'pending'}
# Failure class: WRONG_VALUE
# Diagnosis: Status field depends on internal queue state not initialised
# in this context. The queue is populated by a background thread
# that does not run during synchronous test execution.
# To resolve: Pass status='ok' explicitly or mock the queue initialisation.
"""
...
The TODO comment goes in the exact position where the failing doctest would have
appeared. This preserves the logical structure of the docstring — a reader can see
what was attempted and what would be needed to make it work.
Indentation rules
Indentation errors introduced during injection are syntax errors. Follow these
rules exactly.
Function body indentation
The indentation of the generated body must match the file's convention. Detect the
convention from the existing code in the prompt:
Count the leading spaces on the first indented line visible in the prompt
If that line uses 4 spaces, the body uses 4 spaces per level
If it uses 2 spaces, use 2 spaces
If it uses tabs, use tabs — do not convert
The body's top-level lines (direct children of the function) are indented one level
deeper than the def keyword. If def is at column 0 (module-level function), the
body starts at column 4 (or whatever the detected indent width is). If def is at
column 4 (method inside a class), the body starts at column 8.
# Module-level function — def at column 0, body at column 4
def process(data):
result = []
for item in data:
result.append(item * 2)
return result
# Class method — def at column 4, body at column 8
class Processor:
def process(self, data):
result = []
for item in data:
result.append(item * 2)
return result
Doctest indentation
Doctests inside a docstring are indented to match the docstring body. If the
docstring body is at 4 spaces (standard for a module-level function), the >>>
lines are also at 4 spaces. The expected output lines follow the same indentation.
def example():
"""Example function.
>>> example() ← 4 spaces before >>>
42 ← 4 spaces before expected output
"""
For a class method where the docstring body is at 8 spaces:
Follow this exact sequence to assemble the final output.
FUNCTION_BODY assembly
1. Take the original `prompt` string verbatim — do not modify any character in it
2. IF INJECT_DOCTESTS = true:
a. Find the closing """ of the docstring in the prompt
b. Insert the doctest block (passing tests + TODO comments for failing)
before the closing """
c. Ensure there is exactly one blank line between the last prose line
and the first >>> line
d. Ensure the closing """ is on its own line at the correct indentation
3. Append a single newline after the (modified or unmodified) closing """
IF one is not already present
4. Append the generated function body
— each line already at the correct indentation from the recovery loop
5. The result is the complete reconstructed file content:
prompt_with_doctests + "\n" + function_body
LINE assembly
1. Take the original `prompt` string verbatim
2. Concatenate the completion directly:
full_output = prompt + completion
No newline between them. No space between them.
The completion is the exact characters that complete the line.
3. The result is the complete reconstructed content.
Verification before delivery
Before producing the final output, verify these properties. These are the checks
from Step 10 expressed as concrete assertions.
For both task types:
# The prompt is unchanged
assert full_output.startswith(prompt), \
"ERROR: prompt content was modified during injection"
# The completion was appended (not prepended or inserted elsewhere)
completion_start = len(prompt)
assert full_output[completion_start:].startswith(first_line_of_completion), \
"ERROR: completion not at the correct insertion point"
For FUNCTION_BODY tasks:
# No mixed indentation
lines = full_output.splitlines()
for i, line in enumerate(lines):
if line.startswith('\t') and ' ' in line[:len(line) - len(line.lstrip())]:
raise AssertionError(f"Mixed tabs/spaces at line {i+1}")
# Docstring still closes properly
assert '"""' in full_output or "'''" in full_output, \
"ERROR: docstring closing delimiter missing"
For LINE tasks:
# No leading newline in completion
assert not completion.startswith('\n'), \
"ERROR: spurious newline at start of line completion"
# No additional lines generated
assert '\n' not in completion.rstrip('\n') or task_type != "LINE", \
"ERROR: LINE completion contains multiple lines"
If any assertion fails, do not deliver. Fix the assembly and re-verify.
Output format
Deliver the final result in this exact structure. All fields are required.
─────────────────────────────────────────────────────────────────────
COMPLETION RESULT
─────────────────────────────────────────────────────────────────────
task_id: <from metadata>
task_type: FUNCTION_BODY | LINE
status: SUCCESS | PARTIAL
iterations_used: <n>
doctests_passing: <n passing> / <n total generated>
─────────────────────────────────────────────────────────────────────
COMPLETION (generated code only):
<the generated code — function body or line remainder>
─────────────────────────────────────────────────────────────────────
FULL OUTPUT (prompt + completion, ready to use):
<the complete reconstructed content>
─────────────────────────────────────────────────────────────────────
ESCALATION: <null | escalation report from recovery_loop.md>
─────────────────────────────────────────────────────────────────────
Status values
SUCCESS — all generated doctests pass in the final delivered code. The
developer-written ground truth tests will now run against this code.
PARTIAL — the recovery loop exhausted MAX_RECOVERY_ITERATIONS without all
doctests passing. The best attempt is delivered. Failing doctests are replaced with
TODO comments in the docstring. The escalation report is populated.
What to tell the user
After delivering the structured output, communicate the result in plain language.
Be specific — a developer needs to know what to do next.
On SUCCESS:
Tell the user:
What tier the function was classified as and why it mattered for doctest strategy
How many iterations the recovery loop needed (and what failure was fixed if > 1)
Which doctest cases were generated and what each one verifies
That their ground truth tests will now run against the injected code
Example:
"Classified as Tier 1 (pure function — no external calls). Generated 3 doctests
covering the happy path, the zero edge case, and the negative-input error raise.
All 3 passed on iteration 2 — the first attempt was missing the guard clause for
negative inputs. The completed code has been injected and is ready for your ground
truth test suite."
On PARTIAL:
Tell the user:
Which doctests passed and which failed
Your diagnosis for each failing case (use the failure class and the diagnosis from
the escalation report)
What the developer would need to do to make the failing cases pass
That the best attempt has been injected and TODO comments mark the unresolved cases
Example:
"Classified as Tier 4 (external dependency — DB write). Tested 3 input validation
paths; all 3 pass. The happy-path doctest could not be resolved in 5 iterations —
the function calls db.commit() which requires a live connection. The validation
code has been injected. The TODO comment in the docstring explains what a mock
fixture would need to provide for full coverage."
What injection never does
Never modifies any character of the original prompt string
Never changes the function signature
Never changes the original docstring prose
Never injects a doctest that the runner has not confirmed as passing
Never adds imports that were not in the original prompt
Never produces output that changes the line count or indentation of code
that existed before the completion point
Never delivers without running the assembly verification checks
