ultimatile Bug-to-Contract
Review findings and bug fixes address symptoms. This skill asks: what implicit specification was violated, and is that specification now tested?
A single fix prevents one bug. A contract test prevents the entire class.
Inputs
This skill works from two kinds of input. The fix-commit lane uses a root on diff-root's consumer contract, halt included; the review-findings lane uses none.
| Input | What to collect |
|---|---|
| Review findings | Actionable findings from any review pass (codex, Copilot, human reviewer) that resulted in code changes. The reviewer's framing of the issue. |
| Fix commits | Branch commits whose subject signals a fix: git log <root-rev>..HEAD --oneline --grep="fix" -i, then git show <sha>. |
When both are available, prefer review findings — a reviewer saying "this doesn't handle column-major layout" is a stronger signal than a commit message.
Core procedure
1. Collect signals
From review findings:
- For each actionable finding that resulted in a code change, note what the issue was and what changed.
From fix commits:
- Examine the diff of each fix commit:
git log <root-rev>..HEAD --oneline --grep="fix" -i git show <commit-sha>
2. Ask the contract question
For each signal, determine:
- What broke? — The specific symptom (wrong output, crash, silent corruption, reviewer complaint, etc.)
- What implicit specification was violated? — The unstated property that users expect to hold. This is the contract. Examples:
- Memory layout should not affect computation results
- Operation order in einsum should not affect results (up to floating-point tolerance)
- Transpose is an involution:
transpose(transpose(A)) == A - Scalar type promotion should be consistent across operations
- Empty inputs should produce empty outputs with correct shape
- Is this contract already tested? — Search for existing tests that verify this property in general, not just the specific case that broke.
3. Search for existing contract tests
Look for tests that verify the identified contract beyond the specific case:
grep -r "test.*layout\|test.*order\|row_major.*column_major" --include="*.rs" tests/
If a test exists that covers the general contract, the contract is already guarded. Report this and stop.
4. Classify the contract
Identifying the category helps write a general test rather than a point fix test.
| Category | Description | Example |
|---|---|---|
| Representation invariance | Result is independent of internal representation choices | Memory layout, stride order, storage format |
| Algebraic identity | Mathematical property that must hold | Involution, associativity, commutativity |
| Structural preservation | Shape, type, or metadata properties preserved through operations | Output shape matches specification, dtype preserved |
| Boundary behavior | Correct handling of edge cases | Empty input, zero-size dimension, single element |
| Semantic equivalence | Different spellings of the same operation produce the same result | einsum("ij,jk->ik", A, B) == matmul(A, B) |
5. Propose the contract test
Write a test (or propose one) that verifies the general contract, not just the specific case that broke. The test should:
- Vary the representation: If it's a layout bug, test with all valid layouts
- Use multiple inputs: Not just the one that triggered the issue
- State the contract explicitly: The test name and/or comment should name the property being verified
6. Check for related contracts
Once a contract is identified, look for related contracts in the same category. If memory layout invariance was violated for GEMM, it could also be violated for SVD, QR, eigendecomposition, solve, etc.
Propose tests for related operations if they are missing.
7. Report
Present to the user:
- The implicit contract that was violated (one sentence)
- Whether it was already tested (yes/no + evidence)
- Proposed contract test (code or description)
- Related contracts that may also be untested
Important principles
- General over specific: A regression test for the exact input that broke is weak. A contract test for the property that was violated prevents the entire bug class.
- Name the contract: If you can't name the implicit specification in one sentence, you haven't understood the issue yet.
- Patterns are escalation-worthy: A cluster of findings in the same category (e.g., three layout-related issues in one review cycle) indicates a systematically untested contract. Escalate this to the user.
- Don't boil the ocean: One contract test per finding is enough. Incremental contract coverage is the goal.