Quality Eval Core

Quality-Eval Core

Shared model for the quality-eval cluster. The TDD, E2E, verification, benchmark, linting, and audit spokes all depend on these conventions — keep them consistent here so no spoke contradicts another about what "done" means.

1. The four gates (the cluster's defining model)

Quality is enforced at four stages, cheapest-and-earliest first. Each gate catches a different failure class; a failure that escapes one is meant to be caught by the next.

Edit ──write-time──> Commit ──commit-time──> CI ──build-time──> Release ──ship-time──> Prod
   plankton-          verification-loop        e2e / windows-      production-audit
   code-quality       tdd-workflow             desktop-e2e
                      ai-regression-testing    browser-qa · benchmark

Rule: push each check to the earliest gate that can catch it. A write-time linter should not be doing a production audit's job, and a production audit should not be the first place a type error is noticed.

2. The evidence ladder

A claim is only as strong as the artifact behind it. Climb the ladder; never report a rung you didn't run.

1. Asserted — "it works." (no value)
2. Ran — a command exited 0; paste the output.
3. Tested — a named test covers the behavior and is green.
4. Measured — a number against a baseline (coverage %, p95 latency, pass rate).
5. Reproduced — green k times (pass@k), or across agents/runs.

Everything below "Ran" is a guess. verification-loop and eval-harness exist to force a claim up this ladder before a PR.

3. pass@k — one green run is not a pass

Anything non-deterministic (agent output, flaky E2E, perf, LLM calls) must be judged by how often it passes, not whether it can:

• pass@1 — succeeded once. Says almost nothing for a stochastic task.
• pass@k — succeeded in ≥1 of k runs (capability ceiling).
• pass rate — fraction of k runs that pass (reliability floor — the number that matters for shipping).

eval-harness defines criteria + computes this for Claude Code tasks; agent-eval uses it to rank agents; e2e-testing/browser-qa use it to separate a real failure from a flake.

4. The AI write-and-review blind spot

When the same model writes code and reviews it, its blind spots are correlated — it approves the bugs it was prone to make. Defenses, in order:

• An independent check the model can't reason its way around: a deterministic test (tdd-workflow, ai-regression-testing), a real browser (browser-qa), a real OS UIA tree (windows-desktop-e2e).
• A gate the model cannot satisfy by editing its own config — treat changes to lint/test/CI config in the same iteration as suppressing a violation as a red flag (plankton-code-quality).
• A second judge for subjective output (a different model/agent — agent-eval).

5. Tooling matrix

6. Shared guardrails

• Evidence over assertion — paste the command output; "Ran/Tested/Measured", never "Asserted".
• pass@k for anything stochastic — report the pass rate, not a lucky single green run.
• Earliest gate wins — catch each failure class at the cheapest stage that can.
• Independent judgment — don't let the thing under test grade itself; break the write-and-review loop.
• No config laundering — a gate satisfied by loosening its own config is not satisfied; flag it.
• State coverage honestly — a passing suite over thin tests is a weaker claim than its green color implies; name the gap.