# Lcb Doctest Driven Development

> Doctest-driven code generation for competitive-programming problems. The orchestrator first designs a quality gate of doctests (happy-path AND self-devised edge cases), verifies their expected values with a reference solution, injects them into the code generation prompt, then runs a reflection recovery loop until the code passes the gate. Use for any single-function / single-program coding task where correctness on unseen edge cases matters. The code-generating model never sees the hidden grading tests.

- Skill: `ibm/lcb-doctest-driven-development` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add ibm/lcb-doctest-driven-development`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ibm/lcb-doctest-driven-development/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: IBM (https://skillmd.com/u/ibm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ibm/lcb-doctest-driven-development

---


# Doctest-Driven Development Skill

This skill turns a plain code generator into a **test-first** one. Instead of asking the
model to solve a problem blind, we build a **quality gate** of doctests up front — including
edge cases the public samples don't cover — and drive generation and repair against it.

A thin deterministic harness executes the phases below; each phase is a small tool in
`tools/`, loaded and run **only when that phase is reached** (progressive disclosure). The
model does all the reasoning (doctests, reference, code, repair); the harness only calls the
model and runs code.

## The five patterns (human-native skill patterns)

1. **Progressive disclosure** — this file stays short; each phase's detail lives in its own
   tool module and is invoked only when needed. Deeper rationale: `references/patterns.md`.
2. **Evidence ledger** — every generated doctest carries a `confidence` (HIGH/MED/LOW) and a
   one-line justification. Only doctests backed by evidence become hard oracles.
3. **Quality gate** — the code is judged by real execution against the doctests, not by vibes.
4. **Recovery loop** — on failure, the model diagnoses the root cause and rewrites, bounded,
   never-regressing.
5. **Doctest generation** — the gate is *authored by the model from the problem statement*,
   covering happy paths and self-devised edge cases.

## Phases (run in order)

| Phase | Tool (load when reached) | What it does | Model call |
|---|---|---|---|
| 1. Classify | (harness) | TYPE A (stdin, empty `starter_code`) vs TYPE B (functional) | no |
| 2. Doctest generation + evidence ledger | `tools/doctest_generator.py` | model writes happy + edge doctests, each with a confidence score | yes |
| 3. Validate doctests | `tools/validate_doctests.py` | model writes a brute-force **reference**, harness repairs it until it passes the public samples, then **recomputes each edge's expected value by running the reference** (drops untrusted). Yields the trusted **oracle** = public samples + verified edges. Fallback: public-only. | yes |
| 4. Code generation | `tools/code_generator.py` | trusted doctests are **injected** into the prompt; model writes code to satisfy them | yes |
| 5. Quality gate | `tools/quality_gate.py` | execute the code against the oracle; report pass count + failing cases | no |
| 6. Recovery loop | `tools/recovery_loop.py` | while failing: show all failing cases, model diagnoses + rewrites; ≤5 attempts, never-regress | yes |

Shared helpers: `tools/llm_client.py` (model-agnostic LiteLLM proxy client) and
`tools/execute.py` (sandboxed stdin/functional execution + output comparison).

## Contamination rule (critical)
The model is given **only** `question_content`, `starter_code`, and the **public** samples.
The reference is written from the statement; edge expected-values come from *executing* that
reference. **Hidden/private tests are never shown to any model call** — they are read only by
the separate grader (`scripts/grade.py`) after generation is complete.

## How it is run
- Skill pipeline: `lcb_doctest/run_skill.py` (orchestrates Phases 1–6).
- Baseline (no skill) pipeline: `lcb_doctest/run_baseline.py` (plain generation).
- Grading (separate process, sees hidden tests): `scripts/grade.py`.