NV Reason CXR Quality Verifier
Purpose
- Used for deterministic second-pass review of an
nv-reason-cxrevidence pack. - Checks source-pack success, fixture case/prompt binding, image readability and hash, model/runtime identity, output response shape, limitation disclosure, and forbidden overreach phrases.
- Manifest I/O: inputs are
nv_reason_cxr_evidence_pack; outputs arenv_reason_cxr_quality_report.
Instructions
- Use this verifier only on an evidence pack directory produced by
skills/nv-reason-cxr. - Run it through
eval_engine/run.pywhen producing verifier evidence. - The verifier entrypoint is
scripts/grade.py; do not replace it with prompt-only review. - Treat a pass as engineering evidence only, not as validation of clinical correctness or model quality.
Available Scripts
| Script | Purpose | Arguments |
|---|---|---|
scripts/grade.py |
Primary verifier entrypoint declared by skill_manifest.yaml. |
EVIDENCE_PACK_DIR |
Prerequisites
- Runtime requirements: Python standard library only.
- The input directory must contain
manifest.json,validation_summary.json, andoutput.jsonfrom annv-reason-cxrevidence pack. - The recorded fixture path and image artifact must be available for full fixture/image binding.
Limitations
- Does not validate chest X-ray interpretation, model correctness, report completeness, or medical actionability.
- Forbidden-phrase checks are regex-based and can miss paraphrases.
- Mock-mode packs prove wrapper and gate wiring; they do not prove live NV-Reason-CXR-3B model behavior.
Troubleshooting
| Error | Cause | Fix |
|---|---|---|
target_skill_matches fails |
The fixture is not an nv-reason-cxr evidence pack. |
Re-run the verifier against the source skill pack directory. |
fixture_loaded fails |
The source pack references a fixture that is unavailable in this checkout. | Regenerate the source pack with a committed synthetic fixture. |
image_sha256_matches fails |
The recorded image artifact changed or is unavailable. | Regenerate the source pack or inspect artifact provenance. |
forbidden_phrases_absent fails |
Output contains treatment, regulatory, patient-directed, or absolute-certainty language. | Fix the source skill prompt/runtime guard and regenerate evidence. |
Example
MOCK_NV_REASON_CXR=1 python eval_engine/run.py skills/nv-reason-cxr \
--fixture skills/nv-reason-cxr/fixtures/synthetic_cxr_input.json \
--out runs/nv_reason_cxr_demo
python eval_engine/run.py verifiers/nv_reason_cxr_quality_v1 \
--fixture runs/nv_reason_cxr_demo \
--out runs/nv_reason_cxr_quality
python verifiers/nv_reason_cxr_quality_v1/scripts/grade.py \
runs/nv_reason_cxr_demo