Ref Verify

ref-verify — Reference Hallucination Guard

The specific failure this skill prevents: LLMs describe paper content from recalled training data rather than from what the abstract actually says. A paper gets attributed findings it doesn't contain, or cited for claims that appear nowhere in the text. The fix is one rule applied consistently:

Every content statement about a paper must come from a live-fetched source — abstract for topline/headline claims, full text for mechanism/implementation/procedural claims — quoted or paraphrased verbatim. If you cannot fetch the required source, say so explicitly — never fill the gap with recalled description, and never let a topical abstract match stand in for a mechanism-level claim.

Mode Decision

Pick the mode before doing any work. The choice controls cost and depth.

User provides DOI(s) for sanity check?
  └─ ≤10 refs → Quick Screen all
  └─ >10 refs → Quick Screen all; Full Audit only MISMATCH/DEAD results

User says "find papers on X" or "cite papers supporting claim Y"?
  └─ Full Audit (searching from scratch requires content verification)

User says "verify/check my reference list" or pre-submission audit?
  └─ ≤5 refs  → Full Audit all
  └─ >5 refs  → Quick Screen all first; Full Audit MISMATCH/DEAD + any ref
                 cited for a specific factual claim

User is writing inline and adds a single citation from memory?
  └─ Quick Screen minimum; Full Audit if citing for a specific claim

The expensive part is Full Audit (5-layer, source-depth evidence fetch). Quick Screen costs ~5s per paper. Only escalate to Full Audit when the task genuinely requires content verification.

Executable Engine

Use this as a skill-level workflow. Do not build or require MCP for this path. The CLI is an execution engine the skill can call from the terminal; the manual protocol remains the fallback and the source of truth for layers the CLI does not yet cover.

CLI Availability Check

Before a DOI-backed check, see whether the executable engine is available:

ref-verify --help

npx skills add does not pip-install the Python CLI. If the console script is unavailable, do not treat that as a verification result.

If the console script is unavailable but the repository source is present, use:

PYTHONPATH=src python3 -m ref_verify.cli --help

If both commands fail, follow the manual fallback protocol below. Do not pretend the CLI ran, and do not invent a result from memory.

CLI-first workflow

Use the CLI first when the user provides a DOI or asks whether a DOI-backed abstract supports a specific factual claim. Current CLI scope is CrossRef metadata verification plus DOI-bound abstract claim checks. Claim checks use CrossRef first, then DOI-bound OpenAlex, Semantic Scholar, and PubMed fallback when CrossRef has no abstract.

CrossRef metadata screen:

ref-verify verify-doi <doi> \
  --title "<provided title>" \
  --first-author <provided-first-author-last-name> \
  --year <provided-year> \
  --json

Source-checkout fallback:

PYTHONPATH=src python3 -m ref_verify.cli verify-doi <doi> \
  --title "<provided title>" \
  --first-author <provided-first-author-last-name> \
  --year <provided-year> \
  --json

verify-doi exits 0 only for PASS; WARN and REJECT return non-zero. Supplying only a DOI checks that CrossRef resolves but is not enough to pass metadata verification.

Route the result:

• PASS: report the metadata match, then continue Layer 4 DOI landing-page resolution and Layer 5 retraction checks when the selected mode requires them.
• WARN: report the missing or weak metadata, then continue manual fallback for the missing layers instead of silently accepting the reference.
• REJECT: stop using that citation as verified; report the mismatch and ask for a corrected DOI or citation unless the user explicitly wants alternatives.

Single claim check against a DOI abstract:

ref-verify check-claim <doi> --claim "<specific factual claim>" --json

Source-checkout fallback:

PYTHONPATH=src python3 -m ref_verify.cli check-claim <doi> --claim "<specific factual claim>" --json

By default, check-claim uses CrossRef first. If CrossRef has no abstract, it tries DOI-bound OpenAlex, Semantic Scholar, and PubMed fallback sources. Use --source crossref, --source openalex, --source semantic-scholar, or --source pubmed for source-specific debugging. Explicit non-CrossRef source selection bypasses CrossRef, so it can isolate an OpenAlex, Semantic Scholar, or PubMed failure.

Route the result:

• ACCEPT: quote the CLI evidence and continue any required manual DOI resolution, second-source existence, and retraction layers.
• WARN or PARTIAL: report what the selected abstract source did and did not support, then continue manual fallback when more layers are required.
• UNVERIFIABLE: no trusted DOI-bound abstract evidence was available from the attempted CLI sources. Continue the manual fallback chain below instead of treating the claim as rejected or supported.

CLI ACCEPT is abstract-level evidence only. If the claim describes a mechanism, implementation, or procedure, do not promote that result to the Full Audit verdict; classify the claim in Layer 3 and obtain the required full-text evidence first.

For JSON output, use abstract_source, source_attempts, and error_code to decide the next step:

• CLAIM_SUPPORTED: explicit abstract support found.
• CLAIM_NOT_EXPLICIT: an abstract was available, but the claim was not explicitly supported.
• CLAIM_AMBIGUOUS: numeric evidence or context exists, but binding is ambiguous.
• NO_ABSTRACT: attempted DOI-bound sources did not provide abstract text.
• DOI_NOT_FOUND: selected source did not find a DOI-bound record.
• DOI_MISMATCH: the primary or explicitly selected DOI-bound record did not match the requested DOI.
• SOURCE_API_ERROR, SOURCE_TIMEOUT, SOURCE_RATE_LIMITED, SOURCE_UNSUPPORTED: source lookup failed, timed out, was rate-limited, or could not be used.

The CLI does not replace Layer 4 DOI landing-page resolution, two-source existence checks, or retraction checks. Continue the manual protocol for those layers when the selected mode requires them.

Quick Screen — metadata + DOI sanity check

Use when the user provides a DOI or full citation and wants a sanity check.

1. Hit CrossRef: https://api.crossref.org/works/{DOI}
2. Compare returned title + first-author last name against what user provided
3. Fetch https://doi.org/{DOI} — confirm it resolves and lands on the right paper
4. Report one line per reference:

Smith et al. (2021) 10.1234/example — PASS (title/author match, DOI resolves)
Jones (2019) 10.5678/other — MISMATCH (CrossRef: Jones & Lee 2019, not Jones alone)
Kim (2023) 10.9999/fake — DEAD DOI

Escalate to Full Audit if: DOI resolves to a different paper, any field mismatches, or user is citing for a specific factual claim.

Full Audit — for literature search or pre-submission check

Use when: searching for papers to support a claim, or doing a final citation sweep.

Run all five layers per paper. The layers are ordered by what they catch — don't skip forward.

Layer 1 — Existence

Search two sources independently:

• CrossRef: https://api.crossref.org/works?query.bibliographic={title+author}&rows=5
• Semantic Scholar: https://api.semanticscholar.org/graph/v1/paper/search?query={title+author}&fields=title,authors,year,externalIds,abstract&limit=5
• arXiv for preprints: https://export.arxiv.org/api/query?search_query=ti:{title}&max_results=3

A paper is confirmed only if titles essentially match and first-author last name agrees across two sources.

• Two-source hit → CONFIRMED
• One-source → SINGLE-SOURCE ⚠ — proceed with caution, note in output
• Zero → NOT FOUND ✗ — stop; report clearly; do not invent a substitute

Layer 2 — Metadata

Extract from confirmed sources and compare: title, all authors (last names), year, journal full name, DOI, volume/pages (mark [NOT IN SOURCE] if absent). If any field differs between sources, show both — do not silently pick one.

Layer 3 — Content Traceability ← most important layer

This is where the skill's core value lies. The goal is not just "does this paper exist" but "does this paper actually contain the claim being attributed to it."

Step 0 — Classify the claim.

Before fetching anything, classify what kind of claim is being verified:

• Topline/headline claim — a finding of the kind abstracts conventionally state (overall result, top-line number, general conclusion). Abstract-only evidence is a legitimate source of truth for this class.
• Mechanism/implementation/procedural claim — describes how something works or was done: a specific mechanism, algorithm, architecture, parameter, experimental step, control condition, or a numeric value tied to a table/figure/method section rather than the summary result. Abstracts routinely omit this content even when the paper supports it. Signal words: "mechanism," "how," "architecture," "algorithm," "implementation," "procedure," "protocol," "parameter," "control group," "ablation," "table," "figure," or any claim more specific than what a one-paragraph summary would state.

If the claim is mechanism/implementation-class, an abstract-only check is not sufficient even when the abstract is topically on point. Fetch the abstract as evidence of existence/topic match, but do not resolve the verdict from the abstract alone — continue to Layer 3b below.

Fetch the abstract using this priority order:

1. CrossRef raw JSON: https://api.crossref.org/works/{DOI} — check the abstract field
2. OpenAlex: https://api.openalex.org/works/doi:{DOI}?mailto={contact_email} — reconstruct abstract_inverted_index
3. Semantic Scholar: append &fields=abstract to your S2 DOI lookup
4. Open-access fallback: https://api.unpaywall.org/v2/{DOI}?email={contact_email} — check is_oa and oa_locations
5. arXiv fallback for preprints: https://export.arxiv.org/api/query?id_list={arxiv_id}
6. PubMed Central for life/bio papers: https://www.ncbi.nlm.nih.gov/pmc/articles/{PMCID}/

For APIs that require a contact email, replace {contact_email} with a real, user-authorized address from configuration. Do not send a placeholder address.

After fetching, check: does the abstract contain the specific claim being cited?

• Abstract explicitly contains the claim (quote it verbatim) → CONTENT: SUPPORTED
• Abstract is about the topic but doesn't make the specific claim → CONTENT: PARTIAL — quote what it actually says
• Abstract contradicts the claim → CONTENT: CONTRADICTED — do not use this citation
• Abstract not accessible after trying all sources → CONTENT: UNVERIFIABLE — user must check full text
• Claim is mechanism/implementation-class and full text could not be obtained after trying all Layer 3b sources → CONTENT: ABSTRACT-ONLY — full text unavailable, verification insufficient. This label must never be upgraded to SUPPORTED/ACCEPT.

The rule that cannot be relaxed: if you describe what a paper "shows" or "demonstrates" or "reports," you must quote or directly paraphrase fetched source text at the required depth. Summarizing from memory is not permitted even if you feel confident.

Layer 3b — Full-Text Confirmation (mechanism/implementation-class claims only)

Required whenever Step 0 classified the claim as mechanism/implementation-class, or whenever the abstract verdict is PARTIAL for a claim more specific than the abstract's summary level. Skip only for topline claims that received CONTENT: SUPPORTED directly from the abstract.

Fetch the paper body in this priority order, stopping at the first success:

1. Local Zotero PDF — if a Zotero library is configured, query its database for this DOI, resolve the attachment key, and read the linked PDF. A common default is ~/Zotero/zotero.sqlite with attachments under ~/Zotero/storage/{key}/; discover the actual path instead of assuming it.
2. research-wiki vault — when the research-wiki skill is available, search its configured vault for an already-ingested full-text copy. Do not assume a machine-specific vault path.
3. Open-access link — resolve and fetch the actual full text (not just an availability flag):
   • Unpaywall: https://api.unpaywall.org/v2/{DOI}?email={contact_email} → fetch best_oa_location.url_for_pdf (or any oa_locations[].url_for_pdf).
   • arXiv: https://export.arxiv.org/api/query?id_list={arxiv_id} → fetch the full paper, not only the <summary> field.
4. Failure — if none of the above yields a readable full text, do not treat the abstract as sufficient. Apply the CONTENT: ABSTRACT-ONLY label defined above.

When full text is obtained, quote the passage that supports, partially supports, or contradicts the claim. For a mechanism/implementation claim, CONTENT: SUPPORTED requires a full-text quote — an abstract-level topic match cannot satisfy it.

Layer 4 — DOI Resolution

Fetch https://doi.org/{DOI}. Confirm the landing page matches the expected paper. A 403 (bot-blocked) from a URL slug containing the title and volume is not a dead link — note it as paywalled. A redirect to an unrelated page is a critical failure.

Layer 5 — Retraction

Search "{first author last name}" "{journal name}" retraction and check the DOI landing page for retraction banners. A retracted paper must not be used as a primary source.

Output Format

Quick Screen: one line per reference (see above).

Full Audit: one card per paper, then a summary table.

REFERENCE AUDIT
────────────────────────────────────────────────
Paper:   [Title from live source — not from memory]
DOI:     [DOI] — [✓ Resolves | ✗ Dead | ✗ Wrong paper | ⚠ Paywalled-403]
Authors: [Full list from CrossRef/OpenAlex/S2]
Year:    [Year] — Source: CrossRef | S2 | arXiv
Journal: [Full name]

EXISTENCE:  ✓ Confirmed (sources) | ⚠ Single-source | ✗ Not found
METADATA:   ✓ Consistent | ⚠ Discrepancy: [field: value-A vs value-B]
CONTENT:    ✓ Supported — "[verbatim excerpt — abstract for topline, full text for mechanism claims]"
            ⚠ Partial — source says: "[what it actually says]"
            ⚠ Abstract-only — mechanism/implementation claim, full text unavailable after Zotero/research-wiki/OA attempts
            ✗ Contradicted | — Unverifiable (tried CrossRef/OpenAlex/S2/Unpaywall/arXiv/PubMed)
RETRACTION: ✓ None found | ✗ Retracted

VERDICT: ACCEPT | WARN | REJECT
Reason: [one sentence — what's missing or wrong]
────────────────────────────────────────────────

CONTENT field must show either a verbatim excerpt or an explicit non-support state such as ABSTRACT-ONLY or UNVERIFIABLE — never a summary written from memory.

ACCEPT: two-source confirmed, DOI resolves to the right paper, content supported at the required source depth (abstract for topline claims; full text for mechanism/implementation/procedural claims), and no retraction. WARN: solvable issue — single source, partial content match, abstract inaccessible after trying all fallbacks, or (mechanism/implementation claim) full text inaccessible after trying Zotero/research-wiki/OA fallbacks. The full-text-inaccessible case must carry the ABSTRACT-ONLY label in the summary table (e.g. WARN (ABSTRACT-ONLY: mechanism claim, PDF unreachable)) and can never be reported as ACCEPT. Safe to use if user verifies the flagged item. REJECT: DOI dead or resolves to wrong paper, paper not found anywhere, content contradicted, or retraction confirmed.

Summary table after all cards:

SUMMARY
────────────────────────────────────────────────
1. Smith et al. (2021)  — ACCEPT
2. Kim & Park (2019)    — WARN (abstract unverifiable; try PMC or institutional access)
3. Zhang (2023)         — REJECT (DOI resolves to different paper)
────────────────────────────────────────────────
X / Y verified.  Z need attention.

Anti-Hallucination Rules

• Never recall a DOI from memory — fetch from CrossRef or S2.
• Never describe paper content without fetched source text at the required depth to quote from.
• Never fill in missing metadata by guessing or pattern-matching.
• If two sources disagree, show both — do not choose silently.
• If the source required for the claim depth is inaccessible after all fallbacks, mark UNVERIFIABLE or ABSTRACT-ONLY as defined above — do not substitute a description from memory.
• Never resolve a mechanism/implementation-class claim from an abstract topic match alone — fetch full text via Layer 3b, or mark ABSTRACT-ONLY and stop.

Edge Cases

Preprint vs. published: record both DOIs; prefer published for citation; note if title changed between versions.

Author name variants: "J. Smith" vs "John Smith" — flag but do not merge; let user confirm.

Conference proceedings: volume/pages often absent from CrossRef; mark [NOT IN SOURCE], not guessed.

S2 rate limiting: wait 2s and retry once; if still failing, use CrossRef as primary and note single-source limitation.