Claim Verification

Claim Verification

Overview

Evidence-reliability check that walks every tagged claim in .writing/manuscript/*.tex and proves it supportable. Two core passes — completeness and citation resolution (dual source of truth via Zotero + network, with semantic match against the cited abstract) — plus an optional numeric/table consistency pass. Outputs .writing/verify-report.md per-claim PASS/FAIL and caches resolved DOIs in .writing/verify-cache.json.

The most valuable check for an AI-drafted skeleton is Pass 2: it catches hallucinated DOIs and citations whose abstract does not actually support the claim — exactly the reliability the human author leans on when refining.

Core principle: Evidence before claims, always. Violating the letter of this rule is violating the spirit.

Iron law: No STATUS: verified flip without fresh pass evidence recorded in the report.

Relation to the Claim-First Protocol: during drafting, the claim-first discipline (see ../_shared/core/claim-first-protocol.md) requires resolving a claim's evidence before writing prose against it, do not draft against a stub-status claim. Drafting flips STATUS to evidence_ready when evidence is found. This skill is what flips evidence_ready → verified after the passes succeed. Never edit STATUS to verified manually: the audit trail lives in .writing/verify-report.md.

Optional term-ordering discipline. If the project opted into .writing/glossary.md, drafting must keep every term defined before it is used: a % define: tag belongs in the right section, and no % use: tag may reference a term whose defined_in sits later in the paper. This verification skill does not re-run those checks; they are edit-time invariants, not verification-gate invariants. If you suspect a term ordering drifted after manual edits, sweep manually with grep -n '% \(define\|use\):' .writing/manuscript/*.tex and cross-reference .writing/glossary.md.

When to Use

• The skeleton is ready to hand to the human author (run a final evidence-reliability pass first).
• User requests an interim verification ("check my claims", "verify citations", "run pre-submit checks").
• After a round of manual revisions to the manuscript, to re-confirm evidence before handing the skeleton back to the human author.
• On demand during drafting to check a single section.

Do NOT use during initial drafting to flip claims green — that is drafting's job (it flips stub → evidence_ready, not → verified).

Checklist

Run in order. Stop at first failing pass only if the user requests fail-fast; otherwise collect all failures into the report.

• Confirm .writing/ exists and metadata.yaml is populated (no TODO except YAGNI fields) [inspection]
• Pass 1: Claim Completeness [inspection]
• Pass 2: Citation Resolution (Zotero first, network fallback, semantic match) [attestation] (semantic match is Type-B; see §Type-A vs Type-B below)
• Pass 3: Numeric/Table Consistency (optional; run when tables/numbers are settled) [inspection]
• Optional research-integrity gate (experiment-bearing papers; opt-in), see §Optional research-integrity gate [attestation]
• Write .writing/verify-report.md with per-claim PASS/FAIL [inspection]
• Update .writing/verify-cache.json with resolved DOIs [inspection]
• Flip eligible claim STATUS from evidence_ready to verified (only when its passes succeed: Pass 1 + Pass 2, plus Pass 3 when run) [user-attest] (gated by user confirmation per Step 5)
• Update .writing/progress.md Verification Evidence row with command + exit status + report path [inspection]

Enforceability classes: [inspection] the agent confirms it from its own output; [attestation] the agent ran the procedure but the user owns final confirmation; [user-attest] a user-side rule the agent cannot confirm. Type-A checks are [inspection]; Type-B checks are [attestation] or [user-attest].

Process

Step 0: Preconditions

Before running any pass:

1. Confirm .writing/ exists. If not, abort with instruction to run outlining/drafting first.
2. Read .writing/metadata.yaml. Fail if any top-level key is TODO (except the v1 YAGNI list: multi-author fields). Fail fast — verification is pointless against incomplete metadata.
3. Read .writing/verify-cache.json if present; treat as read-through cache keyed by DOI. Cache entries expire never within a session; user must manually delete to force re-resolution.
4. List all .writing/manuscript/*.tex files and their paired .writing/claims/section_*.md files.

Step 1: Pass 1: Claim Completeness

Goal: every load-bearing paragraph is tagged and backed.

For each .writing/manuscript/*.tex:

1. Parse all ^\s*%\s*claim:\s*(\S+) tags (LaTeX line comments) → set of referenced claim ids.

2. Parse all ^\s*%\s*draft-only markers → record as FAIL unconditionally (draft-only is a drafting escape hatch; it must be resolved before the skeleton is finalized).

3. Parse all [NEEDS-EVIDENCE] literal strings → record as FAIL.

4. For each referenced claim id, locate the matching entry in .writing/claims/section_<NN>_<slug>.md, where <NN>_<slug> is the manuscript file's basename without extension (e.g., manuscript 03_methods.tex pairs with claims section_03_methods.md):

   • Missing entry → FAIL: claim '<id>' referenced in <file> but not defined in claims file.
   • STATUS: stub → FAIL: claim '<id>' still stub; drafting did not resolve EVIDENCE.
   • STATUS: evidence_ready → PASS Pass 1 (it advances through Pass 2, plus Pass 3 when run, to become verified).
   • STATUS: verified → PASS Pass 1 (already verified in prior run; Pass 2 — and Pass 3 when run — may re-validate via cache).

5. For each paragraph that neither has % claim: nor % draft-only: check if section is allow-listed. Any stem ending in _<slug> for slug ∈ {abstract, references, acknowledgments} (the unprotected slugs) is exempt from paragraph-tag enforcement. All other manuscript/NN_*.tex files require every load-bearing paragraph to carry % claim: id or % draft-only. Always-skipped within any file: LaTeX line comments (lines starting with %), blank lines, structural LaTeX commands (lines starting with \section, \subsection, \begin, \end, \label, \caption, etc.). Anything else fails with: paragraph in <file>:<line> lacks % claim: id or % draft-only marker.

6. Abstract citation-free check. For every manuscript/*.tex whose stem ends in _abstract, grep for any LaTeX citation command (\\[a-zA-Z]*cite[a-zA-Z]*\b — covers \cite, \citep, \citet, \nocite, \parencite, \textcite, \autocite, \footcite, \citeauthor, \citeyear, \citealt, \citealp, etc.) and for any % claim: tag. Any hit is a FAIL: abstract must not contain citations or claim tags; found <token> at <file>:<line>. Abstracts must stay citation-free; this catches files that drifted from that rule during manual edits.

Allow-list is configurable. Support .writing/verify-config.yaml with key allowlist_sections: [<filename>, ...] — if present, those filenames extend the default exemption set (the unprotected slugs above) for this skill's checks.

Step 2: Pass 2: Citation Resolution (dual source of truth)

This is the most consequential pass. Implements design.md §14.3 exactly.

For each claim that passed Pass 1 with type: citation EVIDENCE entries:

2a. Zotero-first lookup (if enabled)

1. Read metadata.yaml. If zotero.enabled: true:
   • Run ${CLAUDE_PLUGIN_ROOT}/scripts/check-zotero.sh once per session to confirm credentials and the zotero-mcp binary. Cache result in .writing/findings.md as zotero_verified_at: <ts>.
   • For each citation DOI, call the zotero_search_items MCP tool (from the zotero server in .mcp.json) with query=<DOI>, qmode="everything". If zotero.collection_key is set, filter results to items whose data.collections array contains that key; otherwise keep all hits.
   • Zotero hit (exactly one filtered match): call zotero_get_item_metadata(item_key=<key>) to retrieve the stored abstract. Record source: zotero, zotero_item_key: <key> in the claim EVIDENCE entry. Proceed to 2c (semantic match).
   • Ambiguous (multiple filtered matches for one DOI): log to .writing/findings.md under "Issues" and treat as a miss.
   • Zotero DOI miss, semantic fallback: before giving up on Zotero, call zotero_semantic_search(query=<CLAIM text>, limit=5). When the library has been indexed with fulltext (has_fulltext=True chunks), this matches claim content against paper paragraphs, not just titles/abstracts — catching cases where the DOI in your bibliography is slightly off (preprint vs publisher, pre-print server vs final version) but the paper is actually in the library. For each hit whose similarity_score >= 0.45, call zotero_get_item_metadata(item_key=<key>) and verify by DOI / title match. Exactly one high-confidence match → treat as Zotero hit with source: zotero-semantic, zotero_item_key: <key>, match_score: <float>. Otherwise proceed to 2b.
   • Zotero miss (DOI + semantic): proceed to 2b.
2. If zotero.enabled: false or zotero section absent: skip directly to 2b.

2b. Network fallback

1. Check .writing/verify-cache.json for the DOI. If present AND source field indicates successful prior resolution, use cached abstract_hash to confirm abstract still matches (re-fetch abstract only if hash mismatch or cache entry absent).
2. On cache miss / mismatch, invoke Skill(skill="superpower-writing:citations") with the DOI. This resolves the DOI against Crossref and returns canonical metadata.
3. On failure or ambiguity, invoke Skill(skill="superpower-writing:literature") with the DOI and the CLAIM text. literature queries Crossref (and optionally Semantic Scholar / arXiv / DBLP for CS) and returns abstract + metadata.
4. Network hit: record source: network in the claim EVIDENCE entry.
5. auto_push_new_citations: true behavior: if Zotero is enabled AND auto_push is true AND network (not Zotero) returned the hit, push the resolved item to zotero.collection_key by calling zotero_add_by_doi(doi=<DOI>, collection_key=<key>) from the zotero MCP server. The tool dedupes by DOI internally. Update EVIDENCE source to both and record the returned item key as zotero_item_key.
6. Network miss AND Zotero miss: FAIL: DOI <doi> for claim '<id>' unresolvable via Zotero or Crossref.

2c. Semantic match

Once an abstract is in hand (from Zotero or network):

1. Compute abstract hash and store in .writing/verify-cache.json keyed by DOI: {source, resolved_at, abstract_hash, abstract_excerpt}.
2. Perform an LLM-based semantic match: does the abstract plausibly support the CLAIM text? Use a strict rubric: the claim must not contradict the abstract; the abstract's findings/methods must overlap with the claim's substantive content.
3. Match PASS: record PASS in report with excerpt of supporting abstract sentence.
4. Match FAIL: record FAIL with reason (e.g., "abstract describes CIFAR-10 results; claim is about ImageNet"). This is a soft failure — surface to user for manual review rather than auto-rejecting (semantic match has known FP/FN issues). User confirms or overrides in report before the claim is marked verified.
5. Abstract ambiguous, body lookup: if the abstract neither clearly supports nor contradicts the claim AND source: zotero*, call zotero_semantic_search(query=<CLAIM text>, filters={"parent_item_key": <key>}, limit=3) to surface the three most-relevant chunks from the paper body. Re-run the semantic match against those chunks' matched_text. Only escalate to fetching full body via zotero_get_item_fulltext(item_key=<key>) if the chunk-level check remains ambiguous — fulltext returns the whole paper (often 70K+ chars) and must be read with narrow grep / offset-limit windows, not loaded wholesale into context.

2d. Non-citation EVIDENCE

For EVIDENCE entries with type: dataset, type: figure, type: table, etc.: Pass 2 only validates structural presence of the referenced artifact (dataset identifier exists, figure file exists under .writing/figures/, etc.). Content correctness is user's responsibility.

2e. Optional deep pass — citation-auditor agent

When the user requests a deep audit (--deep flag, or when a thorough citation audit is wanted before handing off the skeleton), dispatch the superpower-writing:citation-auditor agent in a fresh context with the full manuscript and .writing/verify-cache.json. The agent adds six judgment layers Pass 2 does not: over-citation, under-citation, circular/self-citation, staleness, relevance drift (abstract supports claim but not this claim), and seminal-work omission. Its findings are advisory — they merge into verify-report.md under an "Advisory" block, and the user decides whether to act on each item.

Step 3: Pass 3: Numeric/Table Consistency (optional)

Purpose: catch copy-paste drift between prose and tables. Skip during early skeleton stages when tables and numbers are not yet settled; run it once the evaluation numbers stabilize.

1. For each .writing/manuscript/*.tex, extract candidate numeric tokens via regex. Default pattern:

   \b(?:n\s*=\s*)?(?:\d{1,3}(?:,\d{3})+|\d+(?:\.\d+)?)\s*(?:%|‰|p\s*[=<]\s*\d|\(\d+\.\d+[-–]\d+\.\d+\)|M|K)?
   

   Capture n=, percentages, p-values, confidence intervals, plain counts.
2. Build a ground-truth number pool from all tables (LaTeX tabular rows — cells separated by &, rows terminated by \\) across manuscript/*.tex and all figure captions (\caption{...} inside figure environments).
3. For each prose number, confirm it appears verbatim in the ground-truth pool. FAIL otherwise.
4. Support .writing/verify-config.yaml numeric_overrides: [<number>, ...] for narrative numbers that are not table-backed (e.g., round references like "a 2018 benchmark"). Numbers in this list skip the check.
5. Per-claim attribution: a FAIL on number 1,247 inside a paragraph tagged % claim: meth-c1 attaches to claim meth-c1 in the report.

Optional research-integrity gate (experiment-bearing papers)

For papers that report their own experiments, run this OPTIONAL pass alongside Pass 3. Pass 3 confirms prose numbers match tables; the integrity gate goes further and probes whether the numbers reflect reality. It catches a broken pipeline whose constant leaks identically into prose and table, a surprise that may be a bug, a stated seed count that exceeds the result directories on disk, and number-tracing failures (best-seed cherry-pick, delta-arithmetic error, caption-table mismatch, scope overclaim). Each heuristic returns CLEAR, SUSPECTED, or INSUFFICIENT.

Run it only when the user opts in and has experiment artifacts on disk. Skip it for theory, position, and survey papers with no first-party experimental numbers. SUSPECTED is advisory: it surfaces to the user under the report's "Advisory: Research Integrity" block and never auto-rejects, never flips STATUS, never edits prose. See references/research-integrity-gate.md for the heuristics, the three-way verdict contract, and the Type-A/Type-B split.

Step 4: Emit Report

Write .writing/verify-report.md with exact structure:

# Verification Report — <ISO-8601 timestamp>

## Summary
- Pass 1 (Completeness): <N passed> / <M total> claims | <K paragraph-tag failures>
- Pass 2 (Citations): <N resolved> / <M citations> | Zotero: <x> | Network: <y> | Both: <z> | Failed: <f>
- Pass 3 (Numerics, optional): <N verified> / <M numbers extracted> | Overrides: <o> | n/a if not run

## Per-Claim

### claim '<id>' — <PASS | FAIL>
- Pass 1: PASS
- Pass 2: PASS (source: zotero, DOI: 10.xxxx/yyy, matched excerpt: "...")
- Pass 3: PASS (numbers 1247, 0.03 confirmed in Table 1) — or n/a if not run

### claim '<id>' — FAIL
- Pass 2: FAIL — DOI 10.xxxx/zzz unresolvable via Zotero or Crossref
- Action: add citation to Zotero collection '<key>' or fix DOI

### claim '<id>' — SOFT-FAIL (semantic match)
- Pass 2: SOFT-FAIL — abstract describes <X>; claim is about <Y>
- Override: pending   <!-- edit to `overridden: <reason>` to allow the STATUS flip on re-run -->

## Document-Level Failures
(draft-only markers, [NEEDS-EVIDENCE] strings)

- <file>:<line>: `% draft-only` still present

Step 5: Update Claim STATUS

Confirmation gate (mutating step). Steps 5 and the auto_push_new_citations write-back in 2b are the only two actions that mutate state outside the report — STATUS: evidence_ready → verified and pushing a network-resolved citation into the user's Zotero collection. Before performing either, present the full list of eligible claims to flip and citations to push, then get explicit user confirmation. This mirrors the soft-fail override requirement: no green-lit mutation lands silently. If the user declines or wants to inspect first, leave STATUS at evidence_ready and the report on disk — verification is re-runnable and the audit trail is already written. Exception: when the user opened this run with an explicit "verify and flip" / "verify and push" instruction, that standing instruction is the confirmation — do not re-prompt.

For each claim where its passes PASS (Pass 1 + Pass 2, plus Pass 3 when run):

1. Locate the claim entry in its claims/section_*.md file.
2. Update STATUS from evidence_ready to verified.
3. Do not flip STATUS for any claim with a soft-failure semantic match — leave those as evidence_ready pending user confirmation in the report. Once the user edits the report to mark the soft failure as "overridden", re-run verification and the STATUS flip proceeds.

Step 6: Record Verification Evidence

If .writing/ exists, append a row to the Verification Evidence table in .writing/progress.md:

| <timestamp> | claim-verification | .writing/verify-report.md | <pass/fail> | <N> claims verified, <M> failures |

Key Principles

Evidence before claims, always

This skill exists because of failure mode #1: a paper ships with a citation that does not say what the prose claims. Every STATUS: verified flip is an assertion that Pass 2's abstract plausibly supports the claim text. Never flip STATUS without the pass evidence in the report.

Dual source of truth is not redundancy — it's correctness

Zotero stores what the user has vetted. The network stores what the world publishes. A DOI resolvable only on the network but not in Zotero is a citation the user has not personally confirmed — the auto_push_new_citations: true flow closes this gap by writing network hits back to Zotero for future vetting. A DOI present in Zotero but unresolvable on the network is a retracted or moved paper — surface as a warning, do not auto-fail (the user's vetted copy is authoritative).

Cache aggressively, re-fetch cautiously

DOI resolution is expensive and rate-limited. .writing/verify-cache.json keyed by DOI with {source, resolved_at, abstract_hash} lets re-runs complete in seconds. Re-fetch only when: cache absent, abstract_hash mismatch on re-read, or user explicitly forces (e.g., deletes the cache file). Cache file is gitignored (see project .gitignore).

Semantic match is advisory, not gatekeeping

LLM-based semantic match has FP/FN rates that make hard auto-rejection brittle (design.md §11). Treat Pass 2c failures as soft failures: surface in the report, require explicit user override before claim STATUS flips to verified. Do not hide these — a silent false negative is worse than a false positive the user sees.

Verb strength tracks EVIDENCE type

A claim's verb must match the strength of its EVIDENCE entries (see outlining for the type field). Strong evidence (a direct measurement, a reproduced result, a settled citation) earns demonstrate, show, establish. Moderate evidence earns indicate, suggest, support. Limited or associative evidence earns are consistent with, may reflect, raise the possibility. The binding rule: a claim whose EVIDENCE is type: analysis on one workload must not say "establishes". One workload supports "is consistent with" or "suggests", not a general establishment claim. When Pass 1 or the optional integrity gate surfaces a verb that outruns its evidence tier, flag it as advisory for the human author; this skill never rewrites prose.

Type-A vs Type-B

Could a script with no taste answer this gate? Yes, Type-A: the model may self-judge (e.g. a claim tag is present, a DOI resolves, an abstract is citation-free, prose numbers match a table). No, it needs taste, Type-B: route to an independent or cross-model reviewer thread and never reply in place. A loop may DRIVE its own iteration but may not ACQUIT its own Type-B verdict. Pass 1 completeness, Pass 2 DOI resolution, and Pass 3 number matching are Type-A. Pass 2c semantic match and the taste-bearing integrity heuristics are Type-B; route them per planning-foundation/references/review-loop-protocol.md rather than self-clearing.

Fail-loud on missing metadata

Verification depends on metadata.yaml being complete. If claim-verification encounters TODO fields, abort immediately rather than running downstream passes. Verification against incomplete metadata produces a green report the user cannot trust.

Never auto-edit the manuscript

This skill reads .writing/manuscript/*.tex and writes .writing/claims/*.md STATUS fields and the report. It does not touch manuscript prose. Manuscript changes go through drafting (or the human author's manual edits), which follow the claim-first drafting discipline.

Report is the audit trail

The verify-report.md is the document-of-record for this verification round. It is the evidence that backs the skeleton ("all claims passed citation verification on "). Keep one report per round; when a new round starts, archive old reports under .writing/archive/reports/.

Upstream Skill Contracts

This skill invokes these plugin-local skills via the Skill tool with the superpower-writing: prefix:

A missing skill at invocation time is an unrecoverable error: halt verification and surface the install command.

Integration Points

• drafting skill: it flips stub → evidence_ready as it resolves evidence. This skill flips evidence_ready → verified.