PBench
Outcome mismatch
- Stop claiming completion.
- Ask for user approval to snapshot the failure before repair.
- Select the registered source that produced the transcript:
codexorclaude. Runyk pbench capture --source <registered-source> --yesfrom the subject repository. If the transcript source is not registered, stop and report that direct capture is unsupported. - Repair the user task.
- Read the generated
private/authoring-checklist.mdand follow the single printed recovery or finalization action.
The snapshot records dirty tracked and untracked state as unresolved private candidates. Do not silently treat those edits as replay input.
Explicit benchmark operation
- Headless:
yk pbench run --case <case> --agent <agent> --profile <label>. - Manual:
yk pbench run --case <case> --manual --profile <label>. - Compare:
yk pbench report --profile <label>.
Current built-in runners are codex and claude. Use stable profile labels for comparisons.
Invariants
- Private material never enters replay input or benchmark-agent context.
- Finalization must pass fresh strict validation of the exact case bundle.