aibot88 Eval Author
Every requirement in delta.md MUST trace to at least one eval. Eval depth scales with risk (§6.3, §8.2).
Categories (§8.3)
| Category | Asks | Typical form |
|---|---|---|
| Functional | Does the system do what the spec says? | YAML cases, BDD .feature, pytest, jest |
| Security | Does the system avoid CWE-mapped weaknesses? | Targeted exploit attempts, fuzz inputs |
| Performance | p95 latency, throughput, resource budgets | Load script + assertion |
| Accessibility | WCAG 2.2 AA criteria for user-facing surfaces | axe-core, manual checklist |
| Drift | Does runtime still match spec? | Schema diffs, telemetry fingerprints |
Threshold floors (§8.2 — implementations MAY raise, MUST NOT lower)
| Risk | Behavior pass rate | Latency p95 | Constitutional |
|---|---|---|---|
| low | ≥95% | — | 0 MUST violations |
| moderate | ≥98% | within 10% of baseline | 0 MUST; SHOULD requires ADR |
| high | ≥99.5% | within 5% of baseline | 0 violations; full CWE Top 25 clear; threat model signed; canary stable |
Process
- For each requirement in
delta.md, propose at least one eval. For high-risk security or PII requirements, propose Functional + Security. - Layout under
.axis/changes/<id>/evals/<category>/<slug>.<ext>. - Author cases in the runner's native syntax (Braintrust, Promptfoo, pytest, jest, .feature).
- For LLM-judged evals: write the rubric and a κ-validation plan against human ratings (§8.3).
- Set the suite-level threshold = max(risk-level floor, project override in
.axis/evals/config.yaml). - Configure re-run policy per §8.3: N ≥ 3 RECOMMENDED at risk high; median pass rate decides.
Anti-patterns
- Eval theater: thresholds set so nothing fails. Counter: κ-validation, quarterly eval-of-evals (§8.3, §D.5).
- One mega-eval covering all five categories — separate them so the report is informative.
- Hand-coded "pass: true" without an executable assertion.
- Latency assertions that don't reference a baseline.
References
- AXIS-26 §8 Verification Model.
- §D.5 Anti-Patterns.