Bio Verify

Bio-Verify — Hardcoded Biological Claim Verification

Systematic sweep of Python scripts and config files for hardcoded biological claims, cross-checked against authoritative APIs. Empirical error rate: ~10-15% of manually-entered constants have detectable errors. Coordinates and ref/alt alleles are the highest-error category. Config JSON files have higher error rates than scripts.

This skill produces an audit report and, after the audit is complete, immutable VerificationEvent records bound to the exact verified file bytes. Fixes are applied only if --fix is specified or the user confirms. Never write the retired bio_verify_state.json store or treat a stale-evidence refresh as verification of hardcoded constants.

Reference files (loaded on demand, not needed for dispatch):

• references/known-issues.md — 17 documented gotchas from prior audits
• references/audit-history.md — prior audit results table (~529 claims, ~13% error rate)

Claim Taxonomy

Scripts and configs embed these claim types, ordered by verification priority:

Workflow

Step 0: Parse arguments

/bio-verify scripts/modal_nutrigenomics.py           # Single file
/bio-verify scripts/blood_*.py                        # Glob
/bio-verify --sweep                                   # All scripts + configs with bio constants
/bio-verify scripts/trait_variant_panel.py --fix       # Verify and auto-fix
/bio-verify --domain blood                            # Only blood group claims

No arguments = --sweep (scan all scripts/*.py AND config/*.json for files containing bio constants).

Step 1: Identify target files

selve CWD: If CWD is selve (has docs/entities/genes/ and src/selve/pgx.py), see references/selve-claim-patterns.md for claim extraction patterns. Run uv run python3 scripts/verify_pgx_consistency.py first (12 local checks, no API calls).

If --sweep, find files containing hardcoded biological claims. Scan both scripts and config JSON:

# Scripts AND config files with variant dicts, gene lists, coordinate tuples, rsIDs
rg -l \
  -g '*.py' -g '*.json' \
  -g '!scripts/modal_utils.py' -g '!scripts/pipeline_stages.py' \
  -g '!scripts/pipeline_cli.py' -g '!**/__pycache__/**' \
  '(rs[0-9]{4,}|chr[0-9XY]+:[0-9]|"ref"|"alt"|ACMG|gnomAD|ClinVar|HDFN|severity|inheritance|"chrom"|"pos"|"gene")' \
  scripts config

Exclude: modal_utils.py (config only), pipeline_stages.py / pipeline_cli.py (DAG metadata), *_benchmark.py (test data), external catalogs (e.g., expansion_hunter_catalog_grch38.json — unmodified Broad catalog).

Config files often have higher error rates than scripts (blood_type_panel.json was 80% wrong in the 2026-03-21 sweep). Prioritize configs with coordinates.

Step 2: Extract claims from each file

Read each target file. Extract hardcoded biological claims by scanning for:

• Variant dicts: Dicts with keys like rsid, chrom/chr, pos, ref, alt, gene
• Gene coordinate tuples: ("chr9", 4985028, 5128183) — gene region definitions
• Gene lists: Lists/sets of gene symbols (usually UPPER_CASE, 3-10 chars)
• Coordinate tuples/strings: chrN:NNNNN patterns, (chrom, start, end) tuples
• Disease associations: "disease": "..." in variant/gene dicts
• Risk/effect claims: Strings containing "Nx risk", "~N% reduced", "odds ratio", "OR=", "RR="
• Classification constants: severity=, inheritance=, classification=, stars=
• Drug interaction claims: "poor metabolizer", "intermediate", "ultrarapid", CPIC levels
• Literature citations: PMIDs, DOIs, author-year references
• Sensitivity/specificity claims: Percentages attributed to papers

For each claim, record: {file, line, type, claimed_value, context_snippet}.

Important: Include actual data samples from the file (first 2-3 entries of any dict/list) in agent prompts. Schema descriptions alone produce wrong field-name assumptions.

Step 3: Route claims to verification domains

Group extracted claims by verification method:

Step 4: Dispatch verification agents

Dispatch parallel subagents, one per domain (or batched if domains are small). Each agent gets:

1. The claim list for its domain (with file paths, line numbers, and actual data samples)
2. Instructions to use specific MCP tools (listed above)
3. Output format: verification table (see below)
4. Instruction to write results to docs/audit/biomedical-fact-check-YYYY-MM-DD/<domain>-verification.md

Agent dispatch template:

CRITICAL: Stop all searching by your 18th tool call and write your synthesis.

Verify the following {domain} claims from the genomics pipeline.
For each claim, check against the authoritative source using the MCP tools listed.

Claims to verify:
{claim_list_with_file_line_context}

For EACH claim, fill in this row:
| rsID/Gene/Item | Claimed value | Verified value | Match? | Source tool | Impact if wrong | Fix |

Use these MCP tools:
{tool_list_for_domain}

IMPORTANT:
- For coordinates: gnomAD r4 is the ground truth for GRCh38. If variants_lookup returns hg19 in the dbSNP field, do NOT compare hg19 to GRCh38.
- For ref/alt: Always verify the GRCh38 reference allele. Strand inversions are common errors.
- For gene names: Check if the gene was renamed (e.g., BCMO1→BCO1). Old names aren't errors if the script's logic works, but flag for update.
- For gene lists: Check COMPOSITION not just count. A list can have the right count but wrong genes.
- For gene regions: Compare Ensembl gene body to claimed start/end. The claimed region should fully contain the gene.
- For citations: Verify the PMID/DOI exists AND matches the claimed content. Watch for sensitivity/specificity/PPV swaps.

Write your full report to: docs/audit/biomedical-fact-check-{date}/{domain}-verification.md
Include tool provenance (which MCP tool returned each answer).

Parallelism: Dispatch as many agents as needed — batch small domains together to keep total agent count reasonable. No fixed cap; use judgment based on MCP server load.

Step 5: Collate results

After all agents complete:

1. Read each docs/audit/biomedical-fact-check-YYYY-MM-DD/<domain>-verification.md
2. Count: total claims checked, errors found, error rate by type
3. Categorize errors by severity:
   • CRITICAL: Wrong position or ref/alt → silent VCF lookup failure (data loss)
   • HIGH: Wrong gene list membership → false negatives in screening. Wrong disease association → incorrect clinical annotation
   • MEDIUM: Outdated gene name, imprecise risk estimate, wrong PMID, gene region partially truncated
   • LOW: Minor gene annotation (MCM6 vs LCT), acceptable shorthand, legacy nomenclature
4. Write SYNTHESIS.md with error summary, root cause analysis, and fix list

Step 6: Fix (if requested)

If --fix or user confirms:

1. Apply fixes grouped by file (one Edit batch per file)
2. After fixing, run ruff check --select F821,F601 on modified files
3. For coordinate/ref/alt fixes: verify the fix by running a re-verification agent to confirm new values are correct GRCh38
4. Commit with: [curation] Fix N hardcoded bio constants — bio-verify audit YYYY-MM-DD
   • Evidence: docs/audit/biomedical-fact-check-YYYY-MM-DD/

Step 7: Re-verify fixes

After applying fixes, dispatch a lightweight re-verification agent to confirm each corrected value is actually correct. This catches transcription errors in the fix itself. Round 4 of the 2026-03-21 sweep found 0 errors in re-verification — fixes from MCP APIs are reliable, but the check is cheap.

Step 8: Decision journal (if errors found)

If errors were found and fixed, write a decision journal entry:

# Bio-Verify Audit YYYY-MM-DD

**Trigger:** /bio-verify sweep
**Scope:** N files, M claims checked
**Errors:** X found (Y% error rate)
**Error types:** [position: N, ref/alt: N, gene name: N, ...]
**Root cause:** [manual entry without cross-check, liftover errors, stale gene names, ...]
**Fix:** Corrected in commit [hash]

Step 9: Emit exact-subject verification events

The retired state-file updater is fail-closed. After the audit exists and each verified asset is at its final bytes, use the canonical producer. First inspect the dry-run plan; then repeat the same command with --apply:

uv run python3 scripts/emit_hardcoded_constants_event.py \
  --asset scripts/generate_pgx_card.py \
  --audit-path docs/audit/biomedical-fact-check-2026-03-24/pgx-card-verification.md \
  --release-id 2026-03-24 \
  --claims-checked 12 \
  --operator-id codex-bio-verify-2026-03-24 \
  --model-id <actual-model-id>

# After checking the planned genesis/supersede relation:
uv run python3 scripts/emit_hardcoded_constants_event.py \
  --asset scripts/generate_pgx_card.py \
  --audit-path docs/audit/biomedical-fact-check-2026-03-24/pgx-card-verification.md \
  --release-id 2026-03-24 \
  --claims-checked 12 \
  --operator-id codex-bio-verify-2026-03-24 \
  --model-id <actual-model-id> \
  --apply

One invocation may repeat --asset for files covered by the same audit and source-version evidence. The producer hashes each current asset, emits one content-addressed event per file plus a terminating batch event, and advances the manifest pointer through the typed callback. Do not hand-edit manifest event IDs.

Finally run just bio-verify-validate, just bio-verify-status --strict, and just bio-verify-tier-audit. For a sweep, also run just bio-verify-queue to show the remaining exact-subject backlog.

Known Issues Summary

17 documented gotchas in references/known-issues.md. The critical ones for dispatch:

• hg19/GRCh38 confusion (#1, #8): variants_lookup returns hg19 in dbSNP field. Use gnomAD r4 for GRCh38 ground truth. Assembly confusion is the #1 error type.
• Gene list composition (#6): Count can match with wrong genes. Always verify composition.
• Config > script error rate (#7): Config JSON files (especially with coordinates) have higher error rates.
• Effect allele AF convention (#17): Some scripts track effect/archaic allele AF, not gnomAD alt AF. Check convention before flagging.
• CPIC API over web search (#11): Use api.cpicpgx.org directly for PGx pairs. Perplexity is unreliable here.
• GenCC via API only (#12): Web search fabricates GenCC classifications. Use curation_gene_validity or thegencc.org.

Full list with examples and workarounds: references/known-issues.md

Stale-Absence Detection

When verifying claims against APIs, distinguish three outcomes:

This prevents the blind spot where a database removes a sequencing artifact and bio-verify silently preserves a stale Tier 2 state.

When --sweep encounters a 404 for a previously-verified entity:

1. Check a second source (myvariant → ClinVar, or vice versa) to confirm removal
2. If both return 404: mark status: refuted with refutation_source and date
3. If only one returns 404: mark conflict: true and flag for human review

Complementary Checks

Bio-verify checks data correctness (are biological constants accurate?). Two complementary systems check logic correctness:

• Canary gate (just canary in genomics) — 55 sentinel variants through auto_classify(). Catches classification logic regressions. Runs as pre-commit hook on classification file changes.
• Calibration canaries (meta/scripts/calibration-canary.py) — 47 boolean questions testing model calibration. Weekly via /maintain. Not genomics-specific but measures the trustworthiness of LLM judgments used elsewhere in the pipeline.