SCI Result Auditor
Audit trustworthiness, traceability, and consistency. Follow project-local rules and the evidence hierarchy in sci-research-manager.
Start narrow
Use the supplied claim/ID/artifact and its relevant protocol first; consult handoff, query map, or index only to resolve missing context. Audit selected claims/IDs when possible; state scope before a broad audit. Reuse unchanged verified context, but reopen decisive raw evidence when an exact claim or conflict requires it.
Do not connect to remote hosts during a local/read-only audit unless the user explicitly requests remote verification.
Audit experiment evidence
Check:
- stable ID and canonical card;
- raw result, config, command/run path, code commit, dataset/version, split, seeds, metric definition, baseline, and selection rule;
- reproducibility of reported numbers from the cited artifact;
- protocol compatibility and baseline fairness;
- test-set checkpoint/threshold/method selection;
- required controls and competing explanations;
- separation of supports, does-not-support, and confounds.
Never infer a result from a filename or chat summary.
Audit claims and paper artifacts
Check:
- every claim has evidence and calibrated strength;
- exploratory, diagnostic, trend, source-only, or mismatched evidence is not promoted into a main claim;
- old-route residue and unsupported contribution language are absent;
- figures/tables, manuscript, supplementary material, data/code statements, and public code use canonical compatible results;
- negative boundaries and contrary evidence are disclosed when material.
Audit files and memory
Check that referenced cards/raw paths exist or are marked missing; experiment IDs resolve across indexes and claim maps; current paper status/handoff agree; archived records point to their replacement; and navigation/provenance checks pass.
Use scripts/check_project_consistency.py for the deterministic first pass. It prints a report by default and writes only when --output is explicitly supplied.
Report decisions without repairing silently
For every issue, state severity, evidence, impact, and recommended repair. For each audited result, separate:
supports;does_not_support;claim_strength;- optional
paper_role or placement when the project records it separately;
risk;next_action.
Return findings to the current requester; involve sci-research-manager when a project decision or evidence transition is requested, not merely to close a direct audit. Do not change route state, promote evidence, or repair canonical artifacts during the audit. Do not recommend larger training merely because evidence is weak or confounded; identify the smallest check that resolves the uncertainty.
1---2name: sci-result-auditor3description: Perform read-only consistency and reproducibility audits across experiment cards, raw results, indexes, registries, claim-evidence maps, handoffs, manuscripts, and public artifacts. Use when checking whether claims are supported, metrics and protocols are compatible, results are traceable, or project memory is internally consistent. Return findings to the requester; involve sci-research-manager only for a requested project decision or evidence transition. Repairs are separately authorized, not auditor mutations.4---56# SCI Result Auditor78Audit trustworthiness, traceability, and consistency. Follow project-local rules and the evidence hierarchy in `sci-research-manager`.910## Start narrow1112Use the supplied claim/ID/artifact and its relevant protocol first; consult handoff, query map, or index only to resolve missing context. Audit selected claims/IDs when possible; state scope before a broad audit. Reuse unchanged verified context, but reopen decisive raw evidence when an exact claim or conflict requires it.13Do not connect to remote hosts during a local/read-only audit unless the user explicitly requests remote verification.1415## Audit experiment evidence1617Check:1819- stable ID and canonical card;20- raw result, config, command/run path, code commit, dataset/version, split, seeds, metric definition, baseline, and selection rule;21- reproducibility of reported numbers from the cited artifact;22- protocol compatibility and baseline fairness;23- test-set checkpoint/threshold/method selection;24- required controls and competing explanations;25- separation of supports, does-not-support, and confounds.2627Never infer a result from a filename or chat summary.2829## Audit claims and paper artifacts3031Check:3233- every claim has evidence and calibrated strength;34- exploratory, diagnostic, trend, source-only, or mismatched evidence is not promoted into a main claim;35- old-route residue and unsupported contribution language are absent;36- figures/tables, manuscript, supplementary material, data/code statements, and public code use canonical compatible results;37- negative boundaries and contrary evidence are disclosed when material.3839## Audit files and memory4041Check that referenced cards/raw paths exist or are marked missing; experiment IDs resolve across indexes and claim maps; current paper status/handoff agree; archived records point to their replacement; and navigation/provenance checks pass.4243Use `scripts/check_project_consistency.py` for the deterministic first pass. It prints a report by default and writes only when `--output` is explicitly supplied.4445## Report decisions without repairing silently4647For every issue, state severity, evidence, impact, and recommended repair. For each audited result, separate:4849- `supports`;50- `does_not_support`;51- `claim_strength`;52- optional `paper_role` or placement when the project records it separately;53- `risk`;54- `next_action`.5556Return findings to the current requester; involve `sci-research-manager` when a project decision or evidence transition is requested, not merely to close a direct audit. Do not change route state, promote evidence, or repair canonical artifacts during the audit. Do not recommend larger training merely because evidence is weak or confounded; identify the smallest check that resolves the uncertainty.
godzhiwzz-create