code-completion-NO-PROGRESSIVE-DISCLOSURE

Code Completion with Doctest-Driven Validation

This skill completes missing Python code — either a full function body or the remainder of a truncated line — using a test-first approach: generate doctests that define the contract, generate code that fulfils it, then iteratively fix the code until it passes.

The order matters. Doctests before code forces you to commit to what the function should do before deciding how it does it. Misunderstandings caught at the doctest stage cost one iteration. Misunderstandings caught after five rounds of code generation cost everything. The recovery loop then ensures you never deliver code that fails its own tests.

Single-file skill (Progressive Disclosure removed)

This is the NO-PROGRESSIVE-DISCLOSURE ablation. In the full skill, the doctest generation rules, the recovery-loop logic, and the injection/delivery spec live in separate reference files that are opened ONLY when a step needs them. Here, all of that content is inlined into this one file and is resident from the first token. Nothing is loaded on demand. Read the whole thing before processing instances.

Everything you need is in THIS file:

• the 11-step workflow (below),
• the complete Doctest Generation Guide (Appendix A),
• the complete Recovery Loop Reference (Appendix B),
• the complete Injection / Delivery Reference (Appendix C). The only external artifact is the runner script run_doctest.py, which is executed, not read into context.

Understanding your inputs

You will receive a JSON record in one of two shapes. The single metadata key that differs between them is how you tell the tasks apart — check it explicitly, never infer from the prompt shape.

Function body completion — the prompt ends right after a function's docstring closing """ (or after the signature : if there is no docstring). The body is entirely absent. No pass, no ..., nothing. Generate the complete indented body.

{
  "prompt": "...all code up to and including the closing \"\"\" of the docstring...",
  "metadata": {
    "task_id": "CarperAI--trlx/idx",
    "ground_truth": "    the complete function body\n    ...",
    "fpath_tuple": ["CarperAI_trlx", "trlx", "pipeline", "__init__.py"],
    "context_start_lineno": 0,
    "lineno": 19,
    "function_name": "register_datapipeline"
  }
}

Line completion — the prompt ends mid-line. Generate only the characters that complete that one line. Nothing more — no newline, no next line.

{
  "prompt": "...all code up to the truncation point, ending mid-line...",
  "metadata": {
    "task_id": "huggingface_diffusers/0",
    "ground_truth": "    StableDiffusionInpaintPipelineLegacy,",
    "fpath_tuple": ["huggingface_diffusers", "tests", "pipelines", "test_stable_diffusion_inpaint_legacy.py"],
    "context_start_lineno": 0,
    "line_no": 28
  }
}

Routing rule: "function_name" in metadata → FUNCTION_BODY task. "line_no" in metadata → LINE task. These are mutually exclusive.

Configuration

The workflow

Steps run in strict order. Each produces something the next depends on.

Step 1  → Analyse the file
Step 2  → Route to task type
Step 3  → Load additional context (only what is needed)
Step 4  → Classify the target function
Step 5  → Pre-generation quality gate
Step 6  → Generate doctests            [see Appendix A, below]
Step 7  → Validate doctest structure
Step 8  → Generate first candidate code
Step 9  → Recovery loop                [see Appendix B, below]
Step 10 → Post-loop quality gate
Step 11 → Inject and deliver           [see Appendix C, below]

Step 1 — Analyse the file

Read the entire prompt field before doing anything else. Build a mental model of the codebase so that whatever you generate fits naturally into it.

As you read, note:

• Every import statement and what it provides
• Module-level variables and constants
• Class names, their methods, and inheritance relationships
• Other function names visible in the prompt and what they do
• The naming style in use (snake_case, camelCase — infer from existing code)
• How existing functions handle errors: raise, return None, return a sentinel value?
• What types existing functions return
• Any decorators on the target function
• The exact docstring of the target function, verbatim, if present

Stop here if the prompt is empty or unparseable. Report the error rather than proceeding with no context. The rest of the workflow depends entirely on what you learn here.

Step 2 — Route to task type

Check the metadata keys:

• "function_name" present → FUNCTION_BODY task. Record the function name and metadata["lineno"] as the line where the body starts.
• "line_no" present → LINE task. Record metadata["line_no"] as the line being completed.
• Neither present → stop. Report that the task type cannot be determined.

For FUNCTION_BODY: generate everything from the first indented line of the body through the final return or expression. Match the file's indentation (4 spaces standard). Do not alter the signature or the docstring.

For LINE: generate only the characters that come after the last character of the prompt on that same line. No leading newline. No additional lines. Just the remainder of the syntactic construct that was cut off.

Step 3 — Load additional context

The prompt is your primary source. Before loading anything else, exhaust what it already tells you. Only go further if the target function references names that do not appear anywhere in the prompt and cannot be inferred from the import statements.

When you do need more context, use fpath_tuple to identify which files in the same directory or package would define the missing names. Load the minimum needed — one or two files at most.

Never load the repository's test files. Those are the external ground truth that runs after delivery. Loading them contaminates the completion.

Every additional file you load costs context tokens and adds noise. The best completions come from using what is already in front of you well.

Step 4 — Classify the target function

Before generating any doctests, classify the target function into the tier that best describes what it does. The tier determines what kind of doctests are possible and what the recovery loop is actually testing.

Special case for LINE tasks: First determine what syntactic context the truncated line sits in:

• Inside a function body → classify that function using the tier table above
• Inside an import block (from x import () → Tier 5, no doctest possible
• Module-level assignment or constant → Tier 5, no doctest possible
• Class attribute declaration → Tier 5, no doctest possible

For Tier 5 contexts in LINE tasks, skip doctest generation entirely and proceed directly to Step 8. Record that validation was skipped and why.

See Appendix A (inlined below) for detailed guidance on each tier's generation patterns before proceeding to Step 6.

Step 5 — Pre-generation quality gate

Before writing any doctest or any code, confirm you can answer these questions. If you cannot answer one, make the assumption explicit and record your confidence level (high / medium / low). A wrong assumption here propagates through every subsequent step.

• What does this function do? (source: docstring, function name, call sites in the prompt)
• What input types does it accept?
• What does it return, and in what type?
• What exceptions should it raise, and under what conditions?
• Are all imports this implementation will need already present in the prompt?
• For LINE tasks: what syntactic construct is the truncated line completing?

If a critical question has no reasonable answer even after making explicit assumptions, pause and ask rather than guessing silently. Guessing silently on a wrong foundation means the recovery loop spends all its iterations fixing the wrong thing.

Step 6 — Generate doctests

The complete doctest generation rules are inlined below in Appendix A. Apply them directly.

Generate doctests appropriate for the tier you identified in Step 4. The goal is to define the function's contract before implementing it — what goes in, what comes out, what gets raised. If you find yourself unable to write a meaningful expected output, that is a signal you have misclassified the tier, not a reason to write a vague test.

For Tier 1–2 FUNCTION_BODY tasks: generate 2–6 doctests covering:

• The typical happy path with a concrete input and concrete expected output
• At least one edge case (empty input, zero, None, boundary value)
• Any exception the function is documented to raise

For Tier 3 FUNCTION_BODY tasks: generate setup inline in the doctest block before the assertion. Use asyncio.run() for async functions. Instantiate required objects directly in the doctest.

For Tier 4 FUNCTION_BODY tasks: generate doctests only for input validation and error-raise paths. Do not attempt to test the external call itself.

For LINE tasks inside a function (Tier 1–3): generate a doctest for the containing function that exercises the line being completed as part of its normal execution.

All expected output values must be concrete and deterministic. No random output, no memory addresses, no timestamps.

Step 7 — Validate doctest structure

Before running anything, verify the doctests are syntactically correct. A malformed doctest will either silently pass everything (masking bugs) or always fail (burning recovery iterations on a formatting problem rather than a code problem).

Check every doctest block for:

• >>> prefix with exactly one space after the arrows
• ... prefix on continuation lines with exactly one space
• Expected output on the very next line after >>> — no blank line between them
• <BLANKLINE> used wherever expected output contains a blank line
• Exception format is exactly: Traceback (most recent call last): then ... then ExceptionType: message
• No ... as a wildcard in expected output unless # doctest: +ELLIPSIS is on the same >>> line

Fix any formatting problems before proceeding. Do not carry malformed doctests into the recovery loop — the loop assumes the tests are well-formed and diagnoses code failures, not test failures.

Step 8 — Generate first candidate code

Now generate the completion using everything gathered in Steps 1–7:

For FUNCTION_BODY tasks:

• Generate the full body, properly indented from first line to last
• Stay consistent with the imports already in the file — do not introduce new ones unless absolutely necessary and not already importable from what is present
• Honour the naming conventions and error-handling patterns observed in Step 1
• The body must be consistent with the expected outputs you committed to in Step 6
• Do not alter the signature or the docstring

For LINE tasks:

• Generate only the remainder of the truncated line
• Start at the exact character position where the prompt ends
• No leading whitespace, no leading newline
• End at the natural conclusion of the syntactic construct (closing bracket, quote, comma, or nothing if the line ends bare)
• Do not generate the next line

Step 9 — Recovery loop

The complete recovery-loop logic is inlined below in Appendix B. Apply it directly.

Write the candidate code to a temporary file alongside the generated doctests, then run:

python run_doctest.py /tmp/completion_candidate.py

The runner returns structured output showing exactly which doctests passed, which failed, what was expected, and what was actually produced. Use this output to make targeted fixes to the code — not to the doctests. The doctests represent the contract. The code is what changes.

Key rules for the loop:

Track the best attempt across iterations — the candidate with the most passing doctests. If iterations exhaust before all doctests pass, the best attempt is what gets delivered, not the last attempt.

If the same failure repeats after two consecutive fix attempts, stop trying the same approach. Either the function has been misclassified (revisit Step 4 and reclassify) or the doctest expects something the codebase cannot support (revisit Step 6 and adjust the expected value based on what you now know). Do not burn remaining iterations on an approach that has already failed twice.

On exhaustion, inject only the doctests that passed in the best attempt. Replace each failing doctest with a structured TODO comment — never inject a known-failing doctest into the output file. A broken example in a codebase is worse than no example.

Failing doctest replacement format:

# TODO: doctest-completion could not produce a passing example for this case.
# Attempted: MAX_RECOVERY_ITERATIONS iterations
# Last failure:
#   Input:    <what was tested>
#   Expected: <what the doctest expected>
#   Got:      <what the code actually produced>
# To fix: <your diagnosis of what would need to change>

Step 10 — Post-loop quality gate

Before delivering anything, verify:

• Does the completion cover only what was missing — no changes to the existing prompt?
• Is indentation consistent throughout (no mixed tabs/spaces, correct depth)?
• For LINE tasks: does the completion attach cleanly to the last character of the prompt with no spurious leading space or newline?
• Are there syntax errors in the completion (unclosed brackets, mismatched quotes, invalid Python)?
• Does the completion reference any names not in scope (not in imports, not in globals, not in the function's own parameters)? Flag these but do not block on them — they may be inherited attributes or injected by a decorator.
• If retries exhausted: is the TODO replacement ready for each failing doctest?

Block on syntax errors. Flag but proceed on scope warnings.

Step 11 — Inject and deliver

The complete injection/delivery spec is inlined below in Appendix C. Apply it directly.

Assemble the final output:

• The original prompt, unchanged
• The completion appended at the correct position
• If INJECT_DOCTESTS = true: passing doctests embedded in the function's docstring, after the original docstring text and before the closing """
• Failing doctests replaced with TODO comments as specified in Step 9

Deliver in this structure:

task_id:          <from metadata>
task_type:        FUNCTION_BODY or LINE
status:           SUCCESS (all doctests pass) or PARTIAL (best attempt, some failing)
iterations_used:  <n>
doctests_passing: <n of passing> / <n total generated>
completion:       <the generated code only — not the full file>
full_output:      <complete prompt + completion with doctests embedded>
escalation:       null or structured report if PARTIAL

Then tell the user, in plain language:

• What tier the function was classified as and why that mattered
• How many iterations the recovery loop needed
• Which doctests passed and what they verify
• For PARTIAL status: what specifically failed, what the diagnosis is, and what a developer would need to do to resolve it

Be specific. "Skipped DB call testing — validated all 3 input validation paths instead" is useful. "Some tests were skipped" is not.

A worked example

Input (function body task):

{
  "prompt": "def factorial(n):\n    \"\"\"Return n! for non-negative integer n.\"\"\"\n",
  "metadata": {
    "task_id": "example/1",
    "function_name": "factorial",
    "lineno": 2,
    "fpath_tuple": ["example", "math_utils.py"]
  }
}

Step 4: Tier 1 — pure function, deterministic, no external calls.

Step 6 doctests generated:

def factorial(n):
    """Return n! for non-negative integer n.

    >>> factorial(0)
    1
    >>> factorial(5)
    120
    >>> factorial(-1)
    Traceback (most recent call last):
        ...
    ValueError: n must be non-negative
    """

Step 8 first candidate:

    if n < 0:
        raise ValueError("n must be non-negative")
    if n == 0:
        return 1
    return n * factorial(n - 1)

Step 9: Runner reports all 3 doctests pass. Loop exits after iteration 1.

Step 11 output:

task_id:          example/1
task_type:        FUNCTION_BODY
status:           SUCCESS
iterations_used:  1
doctests_passing: 3 / 3

What to tell the user: "Tier 1 function — pure and deterministic, so full input/output doctests were possible. All 3 cases passed on the first attempt: zero input, typical input, and the negative input error path."

What this skill does not do

• Run the developer-written ground truth tests — those execute externally after delivery
• Modify the function signature or existing docstring prose
• Generate completions for multiple functions in one pass
• Load repository test files (doing so contaminates the completion)
• Fix bugs in already-complete code (different task entirely)

APPENDIX A — Doctest Generation Guide (inlined)

This is Appendix A. It is part of this single monolithic skill file (not loaded separately). Its job is to tell you exactly how to generate doctests that are syntactically correct, semantically meaningful, and appropriate for the function you are completing — based on the tier you identified in Step 4.

The doctests you generate here are not the deliverable. They are the validation oracle that the recovery loop uses to judge whether the code you generate is correct. Get them right here and the recovery loop can do its job. Get them wrong and the loop either tests the wrong thing or burns all its iterations on a formatting problem rather than a code problem.

Doctest syntax — the exact rules

These rules come directly from Python's doctest specification. Every generated doctest must follow all of them, no exceptions.

Prompt and continuation lines

>>> single_line_statement
... continuation_of_statement
... more_continuation

• >>> — exactly three > characters followed by exactly one space. Nothing else before the code on that line.
• ... — exactly three . characters followed by exactly one space. Used for any line that continues a multi-line statement (function definitions, if blocks, loops, with blocks, multi-line calls).

Expected output placement

The expected output must appear on the very next line after the final >>> or ... line of a statement. No blank line between the statement and the expected output.

# CORRECT
>>> factorial(5)
120

# WRONG — blank line between statement and expected output
>>> factorial(5)

120

Blank lines inside expected output

If the function's actual output contains a blank line, you cannot use a real blank line in the doctest — a blank line signals the end of expected output. Use <BLANKLINE> instead.

>>> print_with_gap()
first line
<BLANKLINE>
second line

Exception format

Exceptions have a strict three-part structure:

>>> function_that_raises(bad_input)
Traceback (most recent call last):
    ...
ExceptionType: message text here

• The first line must be exactly Traceback (most recent call last): — no variation.
• The middle is exactly ... (four spaces then three dots). This is not the ELLIPSIS option — it is always literal in exception doctests and always works regardless of the ELLIPSIS flag.
• The final line is the exception type and message. This is what doctest actually checks. The traceback body is always ignored.
• The exception message must match exactly unless you add # doctest: +ELLIPSIS and use ... as a wildcard within the message.

Directives

Place directives as comments on the same >>> line they apply to:

>>> some_function()  # doctest: +ELLIPSIS
<SomeObject at 0x...>

>>> another_function()  # doctest: +NORMALIZE_WHITESPACE
a    b    c

>>> expensive_example()  # doctest: +SKIP

Available directives relevant to code completion:

Do not use +SKIP as an escape hatch for laziness. Only use it when the example genuinely cannot run — for example, a function that requires a live server connection where even a stub test is impossible.

What NOT to test directly

Avoid testing these in expected output — they will produce non-deterministic results that fail on different machines or across runs:

• Memory addresses: <MyObject at 0x7f3a...> — use # doctest: +ELLIPSIS
• Dictionary order in Python < 3.7 (not a concern for 3.7+, but be aware)
• Floating point with many decimal places — use round() in the doctest input
• Timestamps, UUIDs, random values — test the type or shape instead
• Any repr() that includes internal state subject to change

Tier-by-tier generation guide

Tier 1 — Pure / Deterministic

The function takes inputs, computes something, returns a result. Same input always gives the same output. No external calls, no state, no I/O.

Generate:

• 2–4 concrete input → output examples
• At least one edge case: zero, empty string, empty list, None where accepted, boundary values
• All documented exception paths

Example — string utility:

def truncate(text, max_length, suffix="..."):
    """Truncate text to max_length, appending suffix if truncated.

    >>> truncate("hello world", 8)
    'hello...'
    >>> truncate("hi", 10)
    'hi'
    >>> truncate("hello world", 8, suffix="—")
    'hello w—'
    >>> truncate("", 5)
    ''
    >>> truncate("hello", 0)
    Traceback (most recent call last):
        ...
    ValueError: max_length must be positive
    """

Example — numeric function:

def celsius_to_fahrenheit(c):
    """Convert Celsius to Fahrenheit.

    >>> celsius_to_fahrenheit(0)
    32.0
    >>> celsius_to_fahrenheit(100)
    212.0
    >>> celsius_to_fahrenheit(-40)
    -40.0
    """

Tier 2 — Contract-Testable

The function's output is non-deterministic (order varies, contains IDs, timing-dependent) but its shape, type, length, or range is predictable.

Generate:

• Tests that check isinstance(result, expected_type)
• Tests that check len(result) or result in valid_range
• Tests that check specific keys exist in a returned dict
• Tests that check invariants that must always hold
• Exception paths remain fully testable

Example — function returning a set or shuffled list:

def unique_words(text):
    """Return the set of unique words in text.

    >>> result = unique_words("the cat sat on the mat")
    >>> isinstance(result, set)
    True
    >>> len(result)
    5
    >>> "cat" in result
    True
    >>> unique_words("")
    set()
    """

Example — function returning a dict with known keys:

def parse_config(config_str):
    """Parse a KEY=VALUE config string into a dict.

    >>> result = parse_config("host=localhost port=8080")
    >>> isinstance(result, dict)
    True
    >>> result["host"]
    'localhost'
    >>> result["port"]
    '8080'
    >>> parse_config("")
    {}
    """

Tier 3 — Setup-Assisted

The function needs context to run: a class instance, a temporary file, an async event loop, or some initial state. The setup must happen inside the doctest block itself.

Generate:

• Inline setup before the function call
• Use asyncio.run() for async functions — do not use await directly in doctests
• Instantiate required objects directly
• Clean up if the setup creates side effects (though doctest isolation usually handles this)

Example — method on a class:

class Counter:
    def __init__(self):
        self.value = 0

    def increment(self, amount=1):
        """Increment the counter by amount.

        >>> c = Counter()
        >>> c.increment()
        >>> c.value
        1
        >>> c.increment(5)
        >>> c.value
        6
        >>> c.increment(-1)
        Traceback (most recent call last):
            ...
        ValueError: amount must be positive
        """

Example — async function:

async def fetch_cached(key, cache):
    """Fetch a value from cache, returning None if missing.

    >>> import asyncio
    >>> cache = {"x": 42}
    >>> asyncio.run(fetch_cached("x", cache))
    42
    >>> asyncio.run(fetch_cached("missing", cache)) is None
    True
    """

Example — function needing a temp file:

def count_lines(filepath):
    """Count the number of lines in a file.

    >>> import tempfile, os
    >>> with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
    ...     _ = f.write("line1\\nline2\\nline3\\n")
    ...     tmp = f.name
    >>> count_lines(tmp)
    3
    >>> os.unlink(tmp)
    """

Tier 4 — Error-Path Only

The function's core behaviour requires external infrastructure: a database connection, a network call, a third-party API, the filesystem at a specific path. Testing the happy path would require mocking the infrastructure, which is outside the scope of inline doctests.

Generate:

• Only the input validation and error-raise paths
• Do not attempt to test the actual external call
• Be explicit in a comment that happy-path testing requires integration setup

Example — function that writes to a database:

def save_user(db_conn, user_id, name):
    """Save a user record to the database.

    # Happy-path testing requires a live database connection.
    # Doctest covers input validation only.

    >>> save_user(None, 1, "Alice")
    Traceback (most recent call last):
        ...
    ValueError: db_conn cannot be None

    >>> save_user("conn", None, "Alice")
    Traceback (most recent call last):
        ...
    ValueError: user_id cannot be None

    >>> save_user("conn", 1, "")
    Traceback (most recent call last):
        ...
    ValueError: name cannot be empty
    """

Example — function that makes an HTTP request:

def get_user_profile(user_id, api_client):
    """Fetch a user profile from the API.

    # Live API testing not possible in doctest.
    # Covers input validation only.

    >>> get_user_profile(None, object())
    Traceback (most recent call last):
        ...
    ValueError: user_id must be a positive integer

    >>> get_user_profile(-1, object())
    Traceback (most recent call last):
        ...
    ValueError: user_id must be a positive integer
    """

Tier 5 — Untestable / Skip

The function cannot be meaningfully tested in a doctest — even the error paths require live external state. Or the completion target is not a function at all (it is an import line, a constant, a class attribute declaration).

Do not generate doctests. Instead, document why:

# DOCTEST SKIPPED — Tier 5
# Reason: <why testing is not possible>
# Context: <what the line/function does>
# To test: <what would be needed for real testing>

Tier 5 triggers for LINE tasks:

• Truncated line is inside an import or from ... import ( block
• Truncated line is a module-level constant: MAX_RETRIES =
• Truncated line is a class attribute: _registry: Dict[str, Any] =
• Truncated line is inside a decorator definition with no callable context
• Truncated line is a type alias: UserID =

In all these cases: proceed directly to Step 8 (code generation). Record in the output that validation was skipped and state the reason.

The doctest block placement rules

For FUNCTION_BODY tasks

Doctests go inside the function's docstring, after the prose description and before the closing """. Follow this structure exactly:

def function_name(args):
    """Original one-line summary.

    Any additional prose from the original docstring goes here,
    preserved verbatim.

    >>> function_name(typical_input)
    expected_output
    >>> function_name(edge_case)
    expected_edge_output
    >>> function_name(bad_input)
    Traceback (most recent call last):
        ...
    ExceptionType: message
    """
    # body goes here

If the original docstring has no prose (it only has the """ close), add a blank line before the first >>>:

def function_name(args):
    """
    >>> function_name(1)
    2
    """

If there is no docstring at all, create one:

def function_name(args):
    """[generated docstring]

    >>> function_name(1)
    2
    """

For LINE tasks (inside a function)

Find the function that contains the truncated line and add doctests to that function's docstring using the same rules as above. If that function has no docstring, create one.

Minimum and maximum doctest counts

More is not always better. A doctest suite with 6 redundant happy-path cases and no edge cases is weaker than one with 3 cases that actually cover the boundaries. Choose cases that would catch the bugs most likely to appear in a naive implementation.

Quick reference — common patterns

Testing a return value directly:

>>> add(2, 3)
5

Testing that something is None:

>>> find_user(999) is None
True

Testing a boolean result:

>>> is_valid_email("user@example.com")
True
>>> is_valid_email("not-an-email")
False

Testing an exception:

>>> divide(1, 0)
Traceback (most recent call last):
    ...
ZeroDivisionError: division by zero

Testing output type:

>>> result = build_index(["a", "b", "c"])
>>> isinstance(result, dict)
True

Testing with floating point (use round to avoid platform variance):

>>> round(area_of_circle(1), 4)
3.1416

Testing a print side effect (function returns None):

>>> print_greeting("Alice")
Hello, Alice!

Testing a multi-line return value:

>>> print(format_table([("name", "age"), ("Alice", 30)]))
name  age
----  ---
Alice 30

Testing with setup state:

>>> stack = Stack()
>>> stack.push(1)
>>> stack.push(2)
>>> stack.pop()
2
>>> stack.pop()
1

APPENDIX B — Recovery Loop Reference (inlined)

This is Appendix B. It is part of this single monolithic skill file (not loaded separately). It governs how the recovery loop runs, how failures are diagnosed, how fixes are targeted, and what happens when iterations are exhausted.

The recovery loop is the core mechanism that separates this skill from a one-shot code generator. Its job is not to retry blindly — it is to read the exact failure output from the runner, understand why the code failed, make a targeted fix that addresses that specific failure without breaking what already passes, and repeat until all doctests pass or the iteration budget is spent.

The loop structure

Before the first iteration, initialise these tracking variables:

current_iteration    = 1
best_attempt         = <the code from Step 8>
best_pass_count      = 0
best_iteration       = 1
final_code           = None          ← stays None until all doctests pass

Then run the loop:

WHILE current_iteration <= MAX_RECOVERY_ITERATIONS:

    1. Write candidate to temp file
    2. Run the test runner
    3. Parse the output
    4. Check results
        → IF all pass: set final_code, break
        → IF some pass: update best_attempt if improved, then diagnose and fix
        → IF none pass: diagnose and fix
    5. Check for repeated failure (same failure twice → reclassify or adjust)
    6. Increment current_iteration

AFTER LOOP:
    → IF final_code is set: proceed to Step 10 (success)
    → ELSE: proceed to Step 10 (partial — use best_attempt)

Step-by-step iteration procedure

1. Write the candidate to a temp file

Write the complete candidate — the original prompt plus the generated completion with doctests embedded — to a temporary file at a consistent path:

/tmp/completion_candidate.py

Overwrite this file on each iteration. The runner uses PID-based module naming internally so it never reads a cached version from a previous iteration.

The file must be a complete, runnable Python file. It must include:

• All imports from the original prompt
• All module-level globals and constants from the original prompt
• The target function with its docstring (including the generated doctests)
• The generated function body

Do not write only the function in isolation — imports and globals are often required for the function to execute at all.

2. Run the test runner

python scripts/run_doctest.py /tmp/completion_candidate.py

Capture the full stdout. The runner will never mix its output with doctest internals — everything you receive is structured and machine-parseable.

3. Parse the output

Read the structured output. Key fields:

STATUS: PASSED | FAILED | ERROR | SKIPPED
TOTAL: <n>          ← total doctests attempted
PASSED: <n>         ← how many passed this iteration
FAILED: <n>         ← how many failed
--- Failure N ---
TEST: <the >>> line that failed>
EXPECTED: <what the doctest expected>
GOT: <what the code produced>
LOCATION: <file>:<line>
---
SUMMARY: <human-readable summary>

STATUS: ERROR means the candidate file has a syntax error or import failure. The code cannot run at all. Treat this as zero passes — diagnose as SYNTAX or IMPORT class failure and fix before anything else.

STATUS: SKIPPED means no doctests were found. This should not happen during the recovery loop (doctests were validated in Step 7). If it does, check that the temp file was written correctly and contains the docstring.

4. Update best attempt tracking

IF PASSED count > best_pass_count:
    best_attempt   = current candidate code
    best_pass_count = current PASSED count
    best_iteration  = current_iteration

Track the best attempt across all iterations, not just the most recent one. The most recent iteration is not necessarily the best — a fix that resolves one failure can accidentally regress another.

5. Check the exit condition

IF STATUS == PASSED (all doctests pass):
    final_code = current candidate
    BREAK — do not run more iterations

Exit as soon as all doctests pass. Do not run extra iterations "just to be sure."

Failure diagnosis

When the runner reports failures, diagnose each one before writing any new code. Making a fix without a diagnosis produces random changes that as often regress passing tests as fix failing ones.

Failure classification

Classify each failure into one of these categories. The category determines the fix strategy.

SYNTAX — The candidate file could not be parsed.

Runner signal: STATUS: ERROR, SUMMARY: SyntaxError: ...

Cause: Generated code has unclosed brackets, mismatched quotes, wrong indentation, invalid Python syntax.

Fix strategy: Read the SyntaxError line number. Fix that specific line. Do not rewrite the entire function — the error is localised. Common sources: missing : after if/ for/def, unclosed ( or [, inconsistent indentation mixing tabs and spaces, f-string with unescaped {.

IMPORT — The candidate imports something not available.

Runner signal: STATUS: ERROR, SUMMARY: Import error — ModuleNotFoundError: ... or ImportError: ...

Cause: Generated code uses a module that is not imported at the top of the file, or uses a name from an import that was not in the original prompt.

Fix strategy: Check the original prompt's import list carefully. If the needed import is present under an alias (e.g., import numpy as np), use the alias. If the import is genuinely missing from the prompt, either find a way to implement without it or add the import — but only if it is a standard library module. Do not add third-party imports that were not in the original prompt.

WRONG_VALUE — The code ran and returned something, but the value was wrong.

Runner signal: STATUS: FAILED, GOT: <something>, EXPECTED: <something else>

Cause: Logic error in the generated code. The implementation computes the wrong result for this input.

Fix strategy: Read the specific TEST line and the EXPECTED vs GOT values. Reason about what the code does with that specific input and where the logic diverges from the expected output. Make the minimal change that corrects the computation for this input without breaking the inputs that already pass.

Common sources: off-by-one errors, wrong operator (// vs /, and vs or), wrong variable referenced, missing case in conditional, incorrect base case in recursion.

WRONG_TYPE — The code returned the right kind of thing but in the wrong type.

Runner signal: STATUS: FAILED, GOT: <something> where the type is visible (e.g., GOT: [1, 2, 3] but EXPECTED: (1, 2, 3))

Cause: Return type does not match what the doctest expects.

Fix strategy: Check the function's type hints and docstring for the intended return type. Wrap or convert the return value. If the doctest uses a type-checking pattern (isinstance(result, list)) and the type is wrong, fix the return type in the code.

MISSING_EXCEPTION — The code should raise an exception but does not.

Runner signal: STATUS: FAILED, EXPECTED: Traceback (most recent call last): ... ExceptionType: message, GOT: <some return value or nothing>

Cause: The input validation or guard clause for this error case is missing or incorrectly placed.

Fix strategy: Add the missing guard. Check the original prompt's docstring — it likely describes the conditions under which exceptions are raised. Add the check before the main logic, not after.

WRONG_EXCEPTION — The code raises an exception, but the wrong one.

Runner signal: STATUS: FAILED, EXPECTED: ValueError: ..., GOT: <exception> TypeError: ... (or similar mismatch)

Cause: Either the wrong exception type is raised, or the exception message does not match.

Fix strategy:

• Wrong type: change raise TypeError(...) to raise ValueError(...) (or whatever the doctest expects)
• Wrong message: the doctest checks the exact message string. Match it precisely.
• If the exception is being raised by a dependency (not by the generated code directly), catch and re-raise with the correct type and message.

UNEXPECTED_EXCEPTION — The code raises an exception when it should not.

Runner signal: STATUS: FAILED, GOT: <exception> SomeError: message, EXPECTED: <a normal return value>

Ca

…(truncated)