Verify Refs

Verify References (Audit-Only)

You help a medical researcher prevent reference hallucinations before submission. This skill audits an existing manuscript or bibliography. It does not write to references/ or manuscript/_src/refs.bib. It does not discover new literature; use /search-lit for discovery and /lit-sync for bib management.

When to Use

• Before journal submission, especially for .docx manuscripts inherited from coauthors or external editors.
• After AI-assisted drafting or revision introduced or modified references.
• When a reviewer or collaborator flags a possibly fabricated citation.
• Before /sync-submission freezes a journal package.

Inputs

1. Manuscript or bibliography path: .md, .docx, .bib, .txt, or .tsv.
2. Optional project root. Default: current working directory.
3. Optional flags passed to the script:
   • --offline: extract and classify references without API verification.
   • --timeout N: HTTP timeout seconds.

Companion: pandoc citation key check

For markdown manuscripts using pandoc [@bibkey] citations, validate citation keys first to catch undefined/unused keys before this audit. If you also use the companion manage-refs skill, run its check_citation_keys.py for this; otherwise use your reference manager's citation-key check.

Then run verify_refs.py against the .bib to validate each entry against PubMed/CrossRef. The two checks are complementary: a citation-key check catches mis-keyed cites; verify_refs.py catches fabricated metadata.

Deterministic Script

Run the bundled script rather than verifying citations by memory:

python "${CLAUDE_SKILL_DIR}/scripts/verify_refs.py" manuscript/manuscript.md --project-root .

For hooks or quick manual runs, use the wrapper:

"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" manuscript/manuscript.md --offline

Manual pre-submission strict run (Phase 1A.5):

"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" manuscript/index.qmd --strict

--strict forbids --offline and exits non-zero on any UNVERIFIED row. Full checkpoint protocol: references/manual_checkpoint_guide.md.

The script uses DOI, PMID, CrossRef, PubMed E-utilities, and OpenAlex where available. If network verification fails, it records UNVERIFIED rather than silently passing.

OpenAlex tertiary index (existence recovery). PubMed covers only biomedical literature and CrossRef's conference-proceedings coverage is uneven, so NeurIPS / ICLR / ACL-style citations — common in medical-AI manuscripts — fall through both and would be marked UNVERIFIED. After the PubMed and CrossRef tiers, the script consults OpenAlex (https://api.openalex.org, free, no API key) only when no authoritative author list was obtained yet (so a reference already resolved by PubMed/CrossRef incurs no extra call). It resolves by DOI when present, otherwise by a title search guarded by a token-similarity threshold so a fabricated title cannot earn a spurious OK. This is the free analogue of the second index (e.g. Scopus) that journal submission portals run alongside CrossRef. OpenAlex display names carry no structured family/given split and mix First Last with Last, First forms, so OpenAlex-sourced authors support an existence check plus a tolerant first-author membership check, but never drive the strict positional or author-count MISMATCH (those stay reserved for PubMed efetch / CrossRef). An OpenAlex miss is recorded as UNVERIFIED, never FABRICATED. Pass --no-openalex to restrict verification to PubMed + CrossRef.

Output Contract (v1.3.0)

v1.2.0 (2026-05) adds duplicate_findings[] to the audit JSON. Verbatim PMID or DOI duplicates within the reference list are flagged as MAJOR findings (resolves /peer-review Phase 2A P7). DOI normalization strips https://doi.org/, http://dx.doi.org/, doi: prefixes plus trailing slashes before comparison so https://doi.org/10.x/abc/ and 10.x/abc collapse to one key. Both submission_safe and fully_verified now require duplicate_findings to be empty.

v1.3.0 (2026-05) extends the author cross-check from first-author-only to the full author list and bumps schema_version to 4. For BibTeX inputs, every cited author family name is compared index-by-index against the authoritative source, and the cited-vs-source author counts are compared. PubMed efetch.fcgi (XML full record) is the truth source when a PMID is present — it is authoritative for given/family names where CrossRef is not (a documented case where CrossRef returned a wrong given name that PubMed efetch corrected). Records now carry cited_authors[], actual_authors[], cited_author_count, and actual_author_count. A correct first author does not establish that the remaining author names are authentic. Plain-text / TSV inputs, which cannot be parsed into a confident full list, degrade gracefully to the first-author check.

Removed in Phase 1A.2 (per docs/artifact_contract.md):

• references/verified_references.tsv — record-level details now live inside reference_audit.json under records[].
• references/library.bib — never this skill's concern. /search-lit produces candidates; /lit-sync (via Better BibTeX) writes manuscript/_src/refs.bib.

Sole-writer enforcement: scripts/validate_project_contract.py will flag any references/* file written by this skill as drift.

Workflow

1. Identify the input file and project root.
2. Run scripts/verify_refs.py.
3. Read qc/reference_audit.json.
4. Report all FABRICATED and MISMATCH rows first (from records[]).
5. Report all duplicate_findings[] entries (verbatim PMID/DOI duplicates — cite renumbering required).
6. If UNVERIFIED rows remain, list them as manual checks and do not call the manuscript fully submission-safe. Rows with note = "pagination_placeholder" (e000–e000 / in press / TBD / forthcoming) need the citation resolved before submission; /self-review Phase 2.5c decides whether any is a P0 blocker.
7. If the user needs a human-readable table, summarize from records[] in chat — do not write a TSV.

Quality Gates

• Gate 1: stop submission if any row is FABRICATED.
• Gate 2: require user confirmation before accepting UNVERIFIED references.
• Gate 3: rerun after any reference edits.
• Gate 4 (added 2026-04-26; extended to full-author in v1.3.0): the cited author list is cross-checked against the authoritative source (PubMed efetch preferred, then CrossRef, then PubMed esummary). A row whose DOI/PMID resolves but whose cited authors do not match — at any index, or in total count — is downgraded to MISMATCH. First-author mismatches get note = "first-author hallucination suspected"; #2..#N family or count mismatches get note = "non-first-author hallucination or count mismatch". This catches the LLM failure mode where a real DOI is paired with invented author names anywhere in the list, not just the lead author. Intentional CSL et-al truncation (cited fewer than source) can be silenced per-entry with a BibTeX _audit_truncated = <N> field.
• Gate 5 (added 2026-05, v1.2.0): PMID/DOI duplicate detection within the reference list. Verbatim duplicates (same PMID or normalized DOI) — a common LLM citation-compilation artifact — are flagged as MAJOR findings in duplicate_findings[]. submission_safe == true requires the list to be empty. Resolves /peer-review Phase 2A P7.
• Gate 6 (added 2026-06): pagination / publication-stage placeholders. A reference whose raw entry still carries e000–e000, in press, TBD, or forthcoming is not yet a fully citable record. Each is marked UNVERIFIED with note = "pagination_placeholder" (a would-be VERIFIED record is downgraded; a worse status is left unchanged). verify-refs is manuscript-agnostic and does not judge centrality — it only flags. The escalation call (is this a method- or headline-load-bearing citation, hence a P0 submission blocker?) is made by /self-review Phase 2.5c, which has the manuscript in hand.

Classification note — citation-metadata confusion is not fabrication. Digits in a DOI suffix sometimes look like a journal article number but differ from the real one (e.g., a DOI tail "77196" against article number 26068, or a "60466-1" suffix against article 6274). This is cosmetic metadata confusion, not a fabricated reference: do not record such rows as FABRICATED when the DOI/PMID resolves and the authors match. A genuine FABRICATED verdict requires a non-resolving identifier or an author cross-check failure (Gate 4), not a mismatch between a DOI suffix and an article number.

Author Cross-Check (Detail)

Two failure patterns motivate the author checks: a real DOI can be paired with the wrong first author, and a correct first author can be followed by fabricated co-author names. DOI resolution and first-author agreement alone cannot verify the full author list.

• The authoritative author list is taken from PubMed efetch.fcgi (XML) when a PMID is present, falling back to CrossRef (DOI) and then PubMed esummary. efetch is preferred because CrossRef is unreliable for given names.
• For BibTeX inputs, the full cited list is parsed (cited_authors[], balanced-brace aware, LaTeX-accent tolerant) and compared family-by-family and by total count against actual_authors[].
• Comparison is tolerant: case, diacritics (NFKD plus Turkish/Polish/Czech/ German/Nordic special letters), hyphen vs space, and name particles ("von", "van", "de", ...) are normalized before matching.
• If the cited authors cannot be parsed confidently, the check degrades to the first-author surname comparison, and if even that is empty it is skipped silently — no false MISMATCH from formatting ambiguity.
• Title-only PubMed search does not return an authoritative author and is therefore excluded from this check.
• Intentional truncation (a bib that cites only the first author, or first five
  • et al., by design) would otherwise trip the count check; mark such entries with _audit_truncated = <N> to downgrade the count mismatch to a note.

Claim Fidelity — does the source say what you say it says?

verify_refs.py answers whether a reference is real and whose it is. It cannot answer whether the sentence citing it is true of it, and a citation can be perfectly real while the claim attached to it is not. That gap is where the failure lives: the DOI resolves, the authors match, the reference list renders, and the sentence is still wrong.

scripts/check_claim_fidelity.py checks the claims that have a checkable answer, against full texts you have already downloaded and converted (/fulltext-retrieval produces exactly that layout — it never fetches anything itself):

python3 "${CLAUDE_SKILL_DIR}/scripts/check_claim_fidelity.py" \
  --manuscript manuscript/manuscript.md \
  --fulltext-dir fulltext/ --bib manuscript/_src/refs.bib \
  --out qc/claim_fidelity.json --strict

Only the quote verdict can fail --strict. Everything else is a prompt to go read the source, because paraphrase is legitimate and a gate that blocks on it would be turned off.

Read the "not checked" lines. A citation with no full text on disk is reported as unresolved and never guessed at, and a source whose extracted text is an abstract is reported as too short to judge — absence proves nothing against an abstract. Silence from this detector means "nothing checkable was wrong", which is not the same as "everything is right".

Sentence-level source evidence table

The same qc/claim_fidelity.json now includes evidence_rows: recognized prose sentence/citation pairs, manuscript coordinates, source-text and PDF hashes, advisory retrieval identity, and a separate assessor-entered comparison. Initial rows are not_assessed, even when bibliographic status is OK and no probe fires.

python3 "${CLAUDE_SKILL_DIR}/scripts/check_claim_fidelity.py" \
  --manuscript manuscript/manuscript.md --bib manuscript/_src/refs.bib \
  --fulltext-dir fulltext/ --retrieval-report pdfs/retrieval_report.json \
  --reference-audit qc/reference_audit.json \
  --out qc/claim_fidelity.json --evidence-table qc/claim_fidelity.md

Inspect the actual source before entering pages, excerpts, metric/unit/denominator, population, direction, and a named assessment. Neither equal numbers nor matching words establish support. Record whether the assessor used AI assistance; do not describe an AI-generated assessment as human approval. Rerun with --reviewed-report qc/claim_fidelity.json to retain annotations. Changed inputs leave old assessments unresolved; unmatched rows remain in the JSON for review. The Markdown table is a derived view, not a second editable evidence store.

See references/claim_evidence_workflow.md for field meanings, re-review steps, source-identity limitations, and the difference between recorded and verified.

What This Skill Does NOT Do

• Does not fetch full texts (use /fulltext-retrieval); claim fidelity reads converted text off disk so it stays deterministic and CI-runnable.
• Does not automatically judge topical fit or semantic support. The probes check limited wording patterns; the evidence table records attributed assessments, not verified facts.
• Does not generate new references from memory.
• Does not replace missing citations with plausible alternatives without /search-lit or user approval.
• Does not sync Zotero collections; use /lit-sync after this audit.

Anti-Hallucination

• Never fabricate titles, DOIs, PMIDs, author lists, journal names, years, volumes, or pages.
• Every OK row must be backed by DOI, PMID, CrossRef, or PubMed title evidence.
• If evidence is unavailable, mark UNVERIFIED and keep it visible.