# Test And Oracle Builder

> Builds tests, evaluation oracles, case matrices, and validation scripts for assignments, labs, exams, and projects in computing. Use when a pedagogical artifact must be corrected reliably and reproducibly.

- Skill: `alainlebret/test-and-oracle-builder` (Agent Skill)
- Install (CLI): `npx skillmds@latest add alainlebret/test-and-oracle-builder`
- Raw SKILL.md: https://api.skillmd.com/api/skills/alainlebret/test-and-oracle-builder/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: alainlebret (https://skillmd.com/u/alainlebret)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/alainlebret/test-and-oracle-builder

---


# Test and Oracle Builder

## Goal

Produce a complete, reproducible test suite that allows a grader (human or automated) to evaluate student submissions without ambiguity. The suite must cover nominal cases, edge cases, and error cases.

## Inputs

Read all of the following before generating any output:
- `mission.json` — level, constraints, evaluation_mode
- `statement.md` — the exam or TP subject that students will receive
- `rubric.md` — grading criteria; tests must be traceable to rubric criteria
- `reference/<solution>` — the reference implementation produced by `reference-code-engineer`

## Output contract

Produce the following artifacts, placed under `tests/`:

| File | Description |
|---|---|
| `tests/test-matrix.md` | Human-readable case matrix: one row per test case with input, expected output, rubric criterion ID, and pass/fail oracle |
| `tests/run_tests.sh` (or `.py`) | Executable script that runs all cases against a student binary and reports results |
| `tests/cases/` | Directory with one subdirectory per test case: `input/`, `expected_output/`, `README.md` |

### `test-matrix.md` format

```markdown
| ID | Description | Input | Expected output | Criterion | Notes |
|----|-------------|-------|-----------------|-----------|-------|
| TC-01 | Nominal: single child process | ... | exit 0, log line present | CR-FORK | ... |
| TC-02 | Edge: empty input | ... | exit 1, error on stderr | CR-ERRHANDLING | ... |
```

## Rules

- Every rubric criterion must be covered by at least one test case.
- Distinguish **oracle tests** (deterministic expected output) from **property tests** (behavioural invariants).
- For C/POSIX assignments: test on Linux; document platform assumptions in `tests/PLATFORM.md`.
- For Python assignments: test with the Python version stated in `mission.json → constraints`.
- Do not rely on fragile output matching (e.g. exact whitespace). Use normalised comparison or regex where appropriate.
- Edge cases must include: empty input, malformed input, resource exhaustion signals (SIGPIPE, SIGTERM), and maximum-size input.
- The oracle script must exit 0 if all tests pass, non-zero otherwise, and emit a structured summary (`tests/results.json`).

## C++ testing conventions
- Prefer `ctest` as the execution entrypoint for automated validation.
- Ensure test instructions work from repository root with:
  - `cmake -S reference -B reference/build`
  - `cmake --build reference/build`
  - `ctest --test-dir reference/build --output-on-failure`
- Include at least one negative/error-path test case in `tests/test-matrix.md`.
- Do not rely on locale-specific output formatting in assertions.

## Java testing conventions
- At minimum, provide a deterministic compile + run validation path.
- Preferred local baseline commands:
  - `javac -d reference/build reference/src/Main.java`
  - `java -cp reference/build Main <sample-input>`
- Ensure tests and sample inputs are self-contained in the repository (no network, no external services).
- Include at least one nominal case and one error-path case in `tests/test-matrix.md`.

## Rust testing conventions
- Provide a deterministic Cargo validation path:
  - `cargo build --manifest-path reference/Cargo.toml`
  - `cargo test --manifest-path reference/Cargo.toml`
- Keep tests deterministic and avoid timing-sensitive assertions.
- Include at least one nominal case and one explicit error-path case in `tests/test-matrix.md`.
- Prefer small, focused unit tests for parsing and core business logic.

## Anti-patterns to avoid

- Tests that only verify the happy path — always test error handling.
- Tests that depend on external network or filesystem state not reproducible in a grading environment.
- Oracles that compare floating-point output without tolerance.
- A test matrix that maps multiple rubric criteria to the same test — keep tests atomic.