Harness Feedback
Treat "the harness is too strict" as an engineering finding, not as permission
to disable a safety check. Find the boundary that owns the mismatch and move
the check to the narrowest profile that actually needs its evidence.
Profiles
Use these profiles unless the project has a more specific, documented contract:
| Profile |
Purpose |
Typical blocking checks |
staging-smoke |
Fast proof that the changed build starts and the critical path works |
build, focused regression, one stable smoke/contract check |
security-proof |
Prove an adversarial or trust-boundary claim |
hostile tests, source/collector proof, fresh-context evaluator |
release-attestation |
Prove the exact releasable artifact and its identity |
signing, Authenticode/tool identity, installer/package checks |
nightly-stress |
Find intermittent and capacity failures |
race, stress, AV/OS matrix, long-running evals |
staging-smoke must not require signing, production credentials, a release
certificate, or a long VM stress run. security-proof may run on an unsigned
staging build when its claim is source or runtime behavior. A release check may
remain blocking for release promotion without becoming a per-edit gate.
Feedback Loop
For every overload signal, record:
- requested profile and change boundary;
- gate that blocked or dominated the run;
- command, elapsed time, failure count, and evidence actually produced;
- whether the gate was relevant, duplicated, flaky, or misplaced;
- the smallest profile split or deletion of duplicate coverage;
- a before/after run of the affected profile and a fresh review of the rule.
Use the deterministic harness-load-advisor.py signal as an intake event. It
stores metadata outside the repository and forces the final report to name the
mismatch. Durable policy changes belong in Git; raw session traces do not.
Required Report
Do not write "overkill" and move on. Report:
Harness feedback: OVERLOAD | CLEAR
Requested profile: staging-smoke | security-proof | release-attestation | nightly-stress
Mis-scoped gate: <name>
Evidence: <command, result, elapsed time, or explicit missing proof>
Correction: <profile split or rule change>
Verification: <before/after commands and result>
Residual risk: <what remains intentionally gated and where>
Gotchas
- A fresh evaluator is an independence control, not a release-signing check.
- A VM can be a reusable execution environment without forcing release identity
checks into every VM smoke.
- A green fast gate does not prove release readiness; a red release-only gate
does not invalidate a staging smoke unless the staging claim depends on it.
- Do not replace a misplaced gate with retries, sleeps, or a bypass marker.
- Do not infer overload from one slow run; distinguish environment failure from
a profile contract error.
Troubleshooting
| Symptom |
Likely cause |
Action |
| Staging smoke asks for signing |
Release gate leaked into staging profile |
split release-attestation and run the smoke on the unsigned staging artifact |
| Security proof blocks on a production VM |
Runtime environment and release identity are coupled |
keep the VM, remove release-only assertions from the security profile |
| Same gate fails repeatedly |
Wrong scope, flaky boundary, or missing fixture |
classify the failure and add a focused reproducer; never silently retry |
| Agent says "tests passed" with no profile |
Evidence contract is incomplete |
require the report fields above and the exact command/result |
| Fix removes a safety check |
Causal ownership was not traced |
restore the check, document the narrower boundary, and re-verify it there |
1---2name: harness-feedback3description: Use when an agent says a test, VM, proof, evaluator, or release gate is overloaded, too strict, blocking staging, or causing false positives; split checks by profile, measure the burden, preserve high-risk evidence, and verify the smallest corrected workflow. Do not use for ordinary test selection, a single test failure, or a full security audit without a harness-scope question.4---56# Harness Feedback78Treat "the harness is too strict" as an engineering finding, not as permission9to disable a safety check. Find the boundary that owns the mismatch and move10the check to the narrowest profile that actually needs its evidence.1112## Profiles1314Use these profiles unless the project has a more specific, documented contract:1516| Profile | Purpose | Typical blocking checks |17|---|---|---|18| `staging-smoke` | Fast proof that the changed build starts and the critical path works | build, focused regression, one stable smoke/contract check |19| `security-proof` | Prove an adversarial or trust-boundary claim | hostile tests, source/collector proof, fresh-context evaluator |20| `release-attestation` | Prove the exact releasable artifact and its identity | signing, Authenticode/tool identity, installer/package checks |21| `nightly-stress` | Find intermittent and capacity failures | race, stress, AV/OS matrix, long-running evals |2223`staging-smoke` must not require signing, production credentials, a release24certificate, or a long VM stress run. `security-proof` may run on an unsigned25staging build when its claim is source or runtime behavior. A release check may26remain blocking for release promotion without becoming a per-edit gate.2728## Feedback Loop2930For every overload signal, record:31321. requested profile and change boundary;332. gate that blocked or dominated the run;343. command, elapsed time, failure count, and evidence actually produced;354. whether the gate was relevant, duplicated, flaky, or misplaced;365. the smallest profile split or deletion of duplicate coverage;376. a before/after run of the affected profile and a fresh review of the rule.3839Use the deterministic `harness-load-advisor.py` signal as an intake event. It40stores metadata outside the repository and forces the final report to name the41mismatch. Durable policy changes belong in Git; raw session traces do not.4243## Required Report4445Do not write "overkill" and move on. Report:4647```text48Harness feedback: OVERLOAD | CLEAR49Requested profile: staging-smoke | security-proof | release-attestation | nightly-stress50Mis-scoped gate: <name>51Evidence: <command, result, elapsed time, or explicit missing proof>52Correction: <profile split or rule change>53Verification: <before/after commands and result>54Residual risk: <what remains intentionally gated and where>55```5657## Gotchas5859- A fresh evaluator is an independence control, not a release-signing check.60- A VM can be a reusable execution environment without forcing release identity61 checks into every VM smoke.62- A green fast gate does not prove release readiness; a red release-only gate63 does not invalidate a staging smoke unless the staging claim depends on it.64- Do not replace a misplaced gate with retries, sleeps, or a bypass marker.65- Do not infer overload from one slow run; distinguish environment failure from66 a profile contract error.6768## Troubleshooting6970| Symptom | Likely cause | Action |71|---|---|---|72| Staging smoke asks for signing | Release gate leaked into staging profile | split `release-attestation` and run the smoke on the unsigned staging artifact |73| Security proof blocks on a production VM | Runtime environment and release identity are coupled | keep the VM, remove release-only assertions from the security profile |74| Same gate fails repeatedly | Wrong scope, flaky boundary, or missing fixture | classify the failure and add a focused reproducer; never silently retry |75| Agent says "tests passed" with no profile | Evidence contract is incomplete | require the report fields above and the exact command/result |76| Fix removes a safety check | Causal ownership was not traced | restore the check, document the narrower boundary, and re-verify it there |
AnastasiyaW