IBM Doctest-Driven Development Skill
This skill turns a plain code generator into a test-first one. Instead of asking the model to solve a problem blind, we build a quality gate of doctests up front — including edge cases the public samples don't cover — and drive generation and repair against it.
A thin deterministic harness executes the phases below; each phase is a small tool in
tools/, loaded and run only when that phase is reached (progressive disclosure). The
model does all the reasoning (doctests, reference, code, repair); the harness only calls the
model and runs code.
The five patterns (human-native skill patterns)
- Progressive disclosure — this file stays short; each phase's detail lives in its own
tool module and is invoked only when needed. Deeper rationale:
references/patterns.md. - Evidence ledger — every generated doctest carries a
confidence(HIGH/MED/LOW) and a one-line justification. Only doctests backed by evidence become hard oracles. - Quality gate — the code is judged by real execution against the doctests, not by vibes.
- Recovery loop — on failure, the model diagnoses the root cause and rewrites, bounded, never-regressing.
- Doctest generation — the gate is authored by the model from the problem statement, covering happy paths and self-devised edge cases.
Phases (run in order)
| Phase | Tool (load when reached) | What it does | Model call |
|---|---|---|---|
| 1. Classify | (harness) | TYPE A (stdin, empty starter_code) vs TYPE B (functional) |
no |
| 2. Doctest generation + evidence ledger | tools/doctest_generator.py |
model writes happy + edge doctests, each with a confidence score | yes |
| 3. Validate doctests | tools/validate_doctests.py |
model writes a brute-force reference, harness repairs it until it passes the public samples, then recomputes each edge's expected value by running the reference (drops untrusted). Yields the trusted oracle = public samples + verified edges. Fallback: public-only. | yes |
| 4. Code generation | tools/code_generator.py |
trusted doctests are injected into the prompt; model writes code to satisfy them | yes |
| 5. Quality gate | tools/quality_gate.py |
execute the code against the oracle; report pass count + failing cases | no |
| 6. Recovery loop | tools/recovery_loop.py |
while failing: show all failing cases, model diagnoses + rewrites; ≤5 attempts, never-regress | yes |
Shared helpers: tools/llm_client.py (model-agnostic LiteLLM proxy client) and
tools/execute.py (sandboxed stdin/functional execution + output comparison).
Contamination rule (critical)
The model is given only question_content, starter_code, and the public samples.
The reference is written from the statement; edge expected-values come from executing that
reference. Hidden/private tests are never shown to any model call — they are read only by
the separate grader (scripts/grade.py) after generation is complete.
How it is run
- Skill pipeline:
lcb_doctest/run_skill.py(orchestrates Phases 1–6). - Baseline (no skill) pipeline:
lcb_doctest/run_baseline.py(plain generation). - Grading (separate process, sees hidden tests):
scripts/grade.py.