Reproducibility First

Reproducibility-First — inner loop (∂L/∂code)

The Metaphor

The trial judge weighing testimony. One witness saying the defendant was at the scene is hearsay — a single data point that could be malice, error, or noise. Two independent witnesses with the same story is evidence. A failure observed once is a rumor; a failure observed twice is a signal. Acting on a single observation is how budget gets burned chasing phantoms.

Overview

One observation is not a gradient. This skill is the first gate on the inner loop of Gradient Descent for Agents — it enforces the noise-suppression invariant before any downstream gradient-consuming skill (root-cause-by-layer, loss-backprop-lens, e2e-driven-iteration) gets a turn. An LLM call is one sample of a distribution. A CI run is one sample of an intermittent fault. A failing test on your machine is one sample of your environment. Acting on a single sample is not SGD — it's noise injection.

Core principle: before you use a failure as the basis for a code edit, you must either (a) reproduce it at least once more, or (b) prove the log is unambiguous signal that does not require repetition.

See ../../docs/ldd/convergence.md §3.3 "Noisy SGD" for why this matters for convergence.

When to Use

Invoke this skill before root-cause-by-layer whenever:

• A test failed once in CI but passed locally (or vice versa)
• An LLM returned a malformed response once
• A log message looks surprising but you can't explain why
• A gate rejected on something that seemed fine in review
• An "it's flaky" narrative is forming around a specific case
• You're about to retry "and see what happens"

The Two Branches

Branch A — Reproduce before editing

Run the failing case at least twice more in an environment as close as possible to the one that failed. Possible outcomes:

"Acceptable flake rate" is not a constant — it's a loss tolerance. For critical paths, 1 of 100 fails already warrants diagnosis. For cosmetic gates, 1 of 10 may be tolerated with a tracking ticket.

Branch B — Unambiguous-signal shortcut

Some logs are load-bearing enough that one sample is already a gradient. The signal must satisfy all three criteria:

1. Deterministic cause. The error message names the cause in a way that cannot be ambiguous: TypeError: 'User' object is not subscriptable at a known code path. There is no "it might also be X" reading.
2. Explains the failure completely. Every part of the observed outcome follows from the named cause. No residual "but why did Y happen" remains.
3. Matches a known contract violation. The cause maps to a contract (type, invariant, precondition) that either does or does not hold — a binary state, not a probabilistic one.

If any one fails, you're not on Branch B; go to Branch A.

Red Flags — STOP, you are about to update on a single sample

• "It passed once, let's ship"
• "It failed once, let me patch"
• "The LLM was bad, retry"
• "CI was flaky, rerun"
• "It's probably [infrastructure], I'll patch defensively"
• "I don't have time to reproduce, just fix it"
• "The log looks like X, must be X" (looks like ≠ is — check Branch B criteria)
• You find yourself editing code based on what you suspect happened, not what the log proves happened

Anti-Patterns

The Contract with root-cause-by-layer

root-cause-by-layer assumes the gradient is real. This skill is the gate that enforces that assumption. The dispatch order is strict:

observe failure
  ↓
reproducibility-first  ←— you are here
  ↓
(Branch A: real signal)  or  (Branch B: unambiguous log)
  ↓
root-cause-by-layer  →  loss-backprop-lens  →  fix

Skipping this skill means every downstream skill operates on possibly-noise. Cheap to apply (1–2 extra runs), expensive to skip (the wrong edit is more expensive than the reproduction).

How to Apply — checklist

1. Log the original observation. Verbatim. Copy the full error / log line. Note the environment (branch, commit, machine, time).
2. Pick the branch. Go through the 3-criterion test for Branch B. If all three hold, document which contract the log violates and proceed to root-cause-by-layer. Otherwise go to Branch A.
3. Branch A: reproduce. Minimum 2 additional runs. Same environment. Record outcomes.
4. Classify. Transient / flaky / deterministic.
5. Act accordingly. Transient → do not edit. Flaky or deterministic → proceed with a real gradient.

Common Rationalizations

Real-World Cue

Any commit message containing fix: flaky X without a linked reproduction (trace, log, second CI run) is a reproducibility-first violation. A proper commit message for a fix following this skill carries evidence: "reproduced 3/5 runs; root cause: race in Y.init".