# Claim Verification > Evidence-reliability check. Walks every % claim tag in LaTeX manuscript/*.tex, confirms \cite{} citekeys resolve against .writing/refs.bib (exported from Zotero), runs semantic match against network-fetched abstracts to catch hallucinated or mismatched citations, optionally checks numeric/table consistency, flags any [NEEDS-EVIDENCE] or draft-only markers. Run on demand or when the skeleton is ready for the human author. - Skill: `sipengxie2024/claim-verification` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add sipengxie2024/claim-verification` - Raw SKILL.md: https://api.skillmd.com/api/skills/sipengxie2024/claim-verification/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: sipengxie2024 (https://skillmd.com/u/sipengxie2024) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/sipengxie2024/claim-verification --- # Claim Verification ## Overview Evidence-reliability check that walks every tagged claim in `.writing/manuscript/*.tex` and proves it supportable. Two core passes — completeness and citation resolution (dual source of truth via Zotero + network, with semantic match against the cited abstract) — plus an optional numeric/table consistency pass. Outputs `.writing/verify-report.md` per-claim PASS/FAIL and caches resolved DOIs in `.writing/verify-cache.json`. The most valuable check for an AI-drafted skeleton is Pass 2: it catches hallucinated DOIs and citations whose abstract does not actually support the claim — exactly the reliability the human author leans on when refining. **Core principle:** Evidence before claims, always. Violating the letter of this rule is violating the spirit. **Iron law:** No `STATUS: verified` flip without fresh pass evidence recorded in the report. **Relation to the Claim-First Protocol:** during drafting, the claim-first discipline (see [`../_shared/core/claim-first-protocol.md`](../_shared/core/claim-first-protocol.md)) requires resolving a claim's evidence before writing prose against it, do not draft against a stub-status claim. Drafting flips `STATUS` to `evidence_ready` when evidence is found. This skill is what flips `evidence_ready` → `verified` after the passes succeed. Never edit `STATUS` to `verified` manually: the audit trail lives in `.writing/verify-report.md`. **Optional term-ordering discipline.** If the project opted into `.writing/glossary.md`, drafting must keep every term defined before it is used: a `% define:` tag belongs in the right section, and no `% use:` tag may reference a term whose `defined_in` sits later in the paper. This verification skill does **not** re-run those checks; they are edit-time invariants, not verification-gate invariants. If you suspect a term ordering drifted after manual edits, sweep manually with `grep -n '% \(define\|use\):' .writing/manuscript/*.tex` and cross-reference `.writing/glossary.md`. ## When to Use - The skeleton is ready to hand to the human author (run a final evidence-reliability pass first). - User requests an interim verification ("check my claims", "verify citations", "run pre-submit checks"). - After a round of manual revisions to the manuscript, to re-confirm evidence before handing the skeleton back to the human author. - On demand during drafting to check a single section. Do NOT use during initial drafting to flip claims green — that is drafting's job (it flips stub → evidence_ready, not → verified). ## Checklist Run in order. Stop at first failing pass only if the user requests fail-fast; otherwise collect all failures into the report. - [ ] Confirm `.writing/` exists and `metadata.yaml` is populated (no `TODO` except YAGNI fields) `[inspection]` - [ ] Pass 1: Claim Completeness `[inspection]` - [ ] Pass 2: Citation Resolution (Zotero first, network fallback, semantic match) `[attestation]` (semantic match is Type-B; see §Type-A vs Type-B below) - [ ] Pass 3: Numeric/Table Consistency (optional; run when tables/numbers are settled) `[inspection]` - [ ] Optional research-integrity gate (experiment-bearing papers; opt-in), see §Optional research-integrity gate `[attestation]` - [ ] Write `.writing/verify-report.md` with per-claim PASS/FAIL `[inspection]` - [ ] Update `.writing/verify-cache.json` with resolved DOIs `[inspection]` - [ ] Flip eligible claim `STATUS` from `evidence_ready` to `verified` (only when its passes succeed: Pass 1 + Pass 2, plus Pass 3 when run) `[user-attest]` (gated by user confirmation per Step 5) - [ ] Update `.writing/progress.md` Verification Evidence row with command + exit status + report path `[inspection]` Enforceability classes: `[inspection]` the agent confirms it from its own output; `[attestation]` the agent ran the procedure but the user owns final confirmation; `[user-attest]` a user-side rule the agent cannot confirm. Type-A checks are `[inspection]`; Type-B checks are `[attestation]` or `[user-attest]`. ## Process ### Step 0: Preconditions Before running any pass: 1. Confirm `.writing/` exists. If not, abort with instruction to run outlining/drafting first. 2. Read `.writing/metadata.yaml`. Fail if any top-level key is `TODO` (except the v1 YAGNI list: multi-author fields). Fail fast — verification is pointless against incomplete metadata. 3. Read `.writing/verify-cache.json` if present; treat as read-through cache keyed by DOI. Cache entries expire never within a session; user must manually delete to force re-resolution. 4. List all `.writing/manuscript/*.tex` files and their paired `.writing/claims/section_*.md` files. ### Step 1: Pass 1: Claim Completeness Goal: every load-bearing paragraph is tagged and backed. For each `.writing/manuscript/*.tex`: 1. Parse all `^\s*%\s*claim:\s*(\S+)` tags (LaTeX line comments) → set of referenced claim ids. 2. Parse all `^\s*%\s*draft-only` markers → record as FAIL unconditionally (draft-only is a drafting escape hatch; it must be resolved before the skeleton is finalized). 3. Parse all `[NEEDS-EVIDENCE]` literal strings → record as FAIL. 4. For each referenced claim id, locate the matching entry in `.writing/claims/section__.md`, where `_` is the manuscript file's basename without extension (e.g., manuscript `03_methods.tex` pairs with claims `section_03_methods.md`): - **Missing entry** → FAIL: `claim '' referenced in but not defined in claims file`. - **STATUS: stub** → FAIL: `claim '' still stub; drafting did not resolve EVIDENCE`. - **STATUS: evidence_ready** → PASS Pass 1 (it advances through Pass 2, plus Pass 3 when run, to become `verified`). - **STATUS: verified** → PASS Pass 1 (already verified in prior run; Pass 2 — and Pass 3 when run — may re-validate via cache). 5. For each paragraph that neither has `% claim:` nor `% draft-only`: check if section is allow-listed. Any stem ending in `_` for slug ∈ {`abstract`, `references`, `acknowledgments`} (the unprotected slugs) is exempt from paragraph-tag enforcement. All other `manuscript/NN_*.tex` files require every load-bearing paragraph to carry `% claim: id` or `% draft-only`. Always-skipped within any file: LaTeX line comments (lines starting with `%`), blank lines, structural LaTeX commands (lines starting with `\section`, `\subsection`, `\begin`, `\end`, `\label`, `\caption`, etc.). Anything else fails with: `paragraph in : lacks % claim: id or % draft-only marker`. 6. **Abstract citation-free check.** For every `manuscript/*.tex` whose stem ends in `_abstract`, grep for any LaTeX citation command (`\\[a-zA-Z]*cite[a-zA-Z]*\b` — covers `\cite`, `\citep`, `\citet`, `\nocite`, `\parencite`, `\textcite`, `\autocite`, `\footcite`, `\citeauthor`, `\citeyear`, `\citealt`, `\citealp`, etc.) and for any `% claim:` tag. Any hit is a FAIL: `abstract must not contain citations or claim tags; found at :`. Abstracts must stay citation-free; this catches files that drifted from that rule during manual edits. **Allow-list is configurable.** Support `.writing/verify-config.yaml` with key `allowlist_sections: [, ...]` — if present, those filenames extend the default exemption set (the unprotected slugs above) for this skill's checks. ### Step 2: Pass 2: Citation Resolution (dual source of truth) This is the most consequential pass. Implements design.md §14.3 exactly. For each claim that passed Pass 1 with `type: citation` EVIDENCE entries: #### 2a. Zotero-first lookup (if enabled) 1. Read `metadata.yaml`. If `zotero.enabled: true`: - Run `${CLAUDE_PLUGIN_ROOT}/scripts/check-zotero.sh` once per session to confirm credentials and the `zotero-mcp` binary. Cache result in `.writing/findings.md` as `zotero_verified_at: `. - For each citation DOI, call the `zotero_search_items` MCP tool (from the `zotero` server in `.mcp.json`) with `query=`, `qmode="everything"`. If `zotero.collection_key` is set, filter results to items whose `data.collections` array contains that key; otherwise keep all hits. - **Zotero hit** (exactly one filtered match): call `zotero_get_item_metadata(item_key=)` to retrieve the stored abstract. Record `source: zotero`, `zotero_item_key: ` in the claim EVIDENCE entry. Proceed to 2c (semantic match). - **Ambiguous** (multiple filtered matches for one DOI): log to `.writing/findings.md` under "Issues" and treat as a miss. - **Zotero DOI miss, semantic fallback:** before giving up on Zotero, call `zotero_semantic_search(query=, limit=5)`. When the library has been indexed with fulltext (has_fulltext=True chunks), this matches claim content against paper paragraphs, not just titles/abstracts — catching cases where the DOI in your bibliography is slightly off (preprint vs publisher, pre-print server vs final version) but the paper is actually in the library. For each hit whose `similarity_score >= 0.45`, call `zotero_get_item_metadata(item_key=)` and verify by DOI / title match. Exactly one high-confidence match → treat as Zotero hit with `source: zotero-semantic`, `zotero_item_key: `, `match_score: `. Otherwise proceed to 2b. - **Zotero miss (DOI + semantic):** proceed to 2b. 2. If `zotero.enabled: false` or `zotero` section absent: skip directly to 2b. #### 2b. Network fallback 1. Check `.writing/verify-cache.json` for the DOI. If present AND `source` field indicates successful prior resolution, use cached abstract_hash to confirm abstract still matches (re-fetch abstract only if hash mismatch or cache entry absent). 2. On cache miss / mismatch, invoke `Skill(skill="superpower-writing:citations")` with the DOI. This resolves the DOI against Crossref and returns canonical metadata. 3. On failure or ambiguity, invoke `Skill(skill="superpower-writing:literature")` with the DOI and the CLAIM text. literature queries Crossref (and optionally Semantic Scholar / arXiv / DBLP for CS) and returns abstract + metadata. 4. **Network hit:** record `source: network` in the claim EVIDENCE entry. 5. **`auto_push_new_citations: true` behavior:** if Zotero is enabled AND auto_push is true AND network (not Zotero) returned the hit, push the resolved item to `zotero.collection_key` by calling `zotero_add_by_doi(doi=, collection_key=)` from the `zotero` MCP server. The tool dedupes by DOI internally. Update EVIDENCE `source` to `both` and record the returned item key as `zotero_item_key`. 6. **Network miss AND Zotero miss:** FAIL: `DOI for claim '' unresolvable via Zotero or Crossref`. #### 2c. Semantic match Once an abstract is in hand (from Zotero or network): 1. Compute abstract hash and store in `.writing/verify-cache.json` keyed by DOI: `{source, resolved_at, abstract_hash, abstract_excerpt}`. 2. Perform an LLM-based semantic match: does the abstract plausibly support the CLAIM text? Use a strict rubric: the claim must not contradict the abstract; the abstract's findings/methods must overlap with the claim's substantive content. 3. **Match PASS:** record PASS in report with excerpt of supporting abstract sentence. 4. **Match FAIL:** record FAIL with reason (e.g., "abstract describes CIFAR-10 results; claim is about ImageNet"). This is a soft failure — surface to user for manual review rather than auto-rejecting (semantic match has known FP/FN issues). User confirms or overrides in report before the claim is marked verified. 5. **Abstract ambiguous, body lookup:** if the abstract neither clearly supports nor contradicts the claim AND `source: zotero*`, call `zotero_semantic_search(query=, filters={"parent_item_key": }, limit=3)` to surface the three most-relevant chunks from the paper body. Re-run the semantic match against those chunks' `matched_text`. Only escalate to fetching full body via `zotero_get_item_fulltext(item_key=)` if the chunk-level check remains ambiguous — fulltext returns the whole paper (often 70K+ chars) and must be read with narrow grep / offset-limit windows, not loaded wholesale into context. #### 2d. Non-citation EVIDENCE For EVIDENCE entries with `type: dataset`, `type: figure`, `type: table`, etc.: Pass 2 only validates structural presence of the referenced artifact (dataset identifier exists, figure file exists under `.writing/figures/`, etc.). Content correctness is user's responsibility. #### 2e. Optional deep pass — `citation-auditor` agent When the user requests a deep audit (`--deep` flag, or when a thorough citation audit is wanted before handing off the skeleton), dispatch the `superpower-writing:citation-auditor` agent in a fresh context with the full manuscript and `.writing/verify-cache.json`. The agent adds six judgment layers Pass 2 does not: over-citation, under-citation, circular/self-citation, staleness, relevance drift (abstract supports claim but not *this* claim), and seminal-work omission. Its findings are advisory — they merge into `verify-report.md` under an "Advisory" block, and the user decides whether to act on each item. ### Step 3: Pass 3: Numeric/Table Consistency (optional) Purpose: catch copy-paste drift between prose and tables. Skip during early skeleton stages when tables and numbers are not yet settled; run it once the evaluation numbers stabilize. 1. For each `.writing/manuscript/*.tex`, extract candidate numeric tokens via regex. Default pattern: ``` \b(?:n\s*=\s*)?(?:\d{1,3}(?:,\d{3})+|\d+(?:\.\d+)?)\s*(?:%|‰|p\s*[=<]\s*\d|\(\d+\.\d+[-–]\d+\.\d+\)|M|K)? ``` Capture `n=`, percentages, p-values, confidence intervals, plain counts. 2. Build a ground-truth number pool from all tables (LaTeX `tabular` rows — cells separated by `&`, rows terminated by `\\`) across manuscript/*.tex and all figure captions (`\caption{...}` inside `figure` environments). 3. For each prose number, confirm it appears verbatim in the ground-truth pool. FAIL otherwise. 4. Support `.writing/verify-config.yaml` `numeric_overrides: [, ...]` for narrative numbers that are not table-backed (e.g., round references like "a 2018 benchmark"). Numbers in this list skip the check. 5. Per-claim attribution: a FAIL on number `1,247` inside a paragraph tagged `% claim: meth-c1` attaches to claim `meth-c1` in the report. ### Optional research-integrity gate (experiment-bearing papers) For papers that report their own experiments, run this OPTIONAL pass alongside Pass 3. Pass 3 confirms prose numbers match tables; the integrity gate goes further and probes whether the numbers reflect reality. It catches a broken pipeline whose constant leaks identically into prose and table, a surprise that may be a bug, a stated seed count that exceeds the result directories on disk, and number-tracing failures (best-seed cherry-pick, delta-arithmetic error, caption-table mismatch, scope overclaim). Each heuristic returns CLEAR, SUSPECTED, or INSUFFICIENT. Run it only when the user opts in and has experiment artifacts on disk. Skip it for theory, position, and survey papers with no first-party experimental numbers. SUSPECTED is advisory: it surfaces to the user under the report's "Advisory: Research Integrity" block and never auto-rejects, never flips `STATUS`, never edits prose. See `references/research-integrity-gate.md` for the heuristics, the three-way verdict contract, and the Type-A/Type-B split. ### Step 4: Emit Report Write `.writing/verify-report.md` with exact structure: ```markdown # Verification Report — ## Summary - Pass 1 (Completeness): / claims | - Pass 2 (Citations): / | Zotero: | Network: | Both: | Failed: - Pass 3 (Numerics, optional): / | Overrides: | n/a if not run ## Per-Claim ### claim '' — - Pass 1: PASS - Pass 2: PASS (source: zotero, DOI: 10.xxxx/yyy, matched excerpt: "...") - Pass 3: PASS (numbers 1247, 0.03 confirmed in Table 1) — or n/a if not run ### claim '' — FAIL - Pass 2: FAIL — DOI 10.xxxx/zzz unresolvable via Zotero or Crossref - Action: add citation to Zotero collection '' or fix DOI ### claim '' — SOFT-FAIL (semantic match) - Pass 2: SOFT-FAIL — abstract describes ; claim is about - Override: pending ## Document-Level Failures (draft-only markers, [NEEDS-EVIDENCE] strings) - :: `% draft-only` still present ``` ### Step 5: Update Claim STATUS **Confirmation gate (mutating step).** Steps 5 and the `auto_push_new_citations` write-back in 2b are the only two actions that mutate state outside the report — `STATUS: evidence_ready → verified` and pushing a network-resolved citation into the user's Zotero collection. Before performing either, present the full list of eligible claims to flip and citations to push, then get explicit user confirmation. This mirrors the soft-fail override requirement: no green-lit mutation lands silently. If the user declines or wants to inspect first, leave STATUS at `evidence_ready` and the report on disk — verification is re-runnable and the audit trail is already written. Exception: when the user opened this run with an explicit "verify and flip" / "verify and push" instruction, that standing instruction is the confirmation — do not re-prompt. For each claim where its passes PASS (Pass 1 + Pass 2, plus Pass 3 when run): 1. Locate the claim entry in its `claims/section_*.md` file. 2. Update `STATUS` from `evidence_ready` to `verified`. 3. Do **not** flip STATUS for any claim with a soft-failure semantic match — leave those as `evidence_ready` pending user confirmation in the report. Once the user edits the report to mark the soft failure as "overridden", re-run verification and the STATUS flip proceeds. ### Step 6: Record Verification Evidence If `.writing/` exists, append a row to the Verification Evidence table in `.writing/progress.md`: ``` | | claim-verification | .writing/verify-report.md | | claims verified, failures | ``` ## Key Principles ### Evidence before claims, always This skill exists because of failure mode #1: a paper ships with a citation that does not say what the prose claims. Every `STATUS: verified` flip is an assertion that Pass 2's abstract plausibly supports the claim text. Never flip STATUS without the pass evidence in the report. ### Dual source of truth is not redundancy — it's correctness Zotero stores what the user has vetted. The network stores what the world publishes. A DOI resolvable only on the network but not in Zotero is a citation the user has not personally confirmed — the `auto_push_new_citations: true` flow closes this gap by writing network hits back to Zotero for future vetting. A DOI present in Zotero but unresolvable on the network is a retracted or moved paper — surface as a warning, do not auto-fail (the user's vetted copy is authoritative). ### Cache aggressively, re-fetch cautiously DOI resolution is expensive and rate-limited. `.writing/verify-cache.json` keyed by DOI with `{source, resolved_at, abstract_hash}` lets re-runs complete in seconds. Re-fetch only when: cache absent, abstract_hash mismatch on re-read, or user explicitly forces (e.g., deletes the cache file). Cache file is gitignored (see project `.gitignore`). ### Semantic match is advisory, not gatekeeping LLM-based semantic match has FP/FN rates that make hard auto-rejection brittle (design.md §11). Treat Pass 2c failures as soft failures: surface in the report, require explicit user override before claim STATUS flips to `verified`. Do not hide these — a silent false negative is worse than a false positive the user sees. ### Verb strength tracks EVIDENCE type A claim's verb must match the strength of its EVIDENCE entries (see outlining for the `type` field). Strong evidence (a direct measurement, a reproduced result, a settled citation) earns `demonstrate`, `show`, `establish`. Moderate evidence earns `indicate`, `suggest`, `support`. Limited or associative evidence earns `are consistent with`, `may reflect`, `raise the possibility`. The binding rule: a claim whose EVIDENCE is `type: analysis` on one workload must not say "establishes". One workload supports "is consistent with" or "suggests", not a general establishment claim. When Pass 1 or the optional integrity gate surfaces a verb that outruns its evidence tier, flag it as advisory for the human author; this skill never rewrites prose. ### Type-A vs Type-B Could a script with no taste answer this gate? Yes, Type-A: the model may self-judge (e.g. a claim tag is present, a DOI resolves, an abstract is citation-free, prose numbers match a table). No, it needs taste, Type-B: route to an independent or cross-model reviewer thread and never reply in place. A loop may DRIVE its own iteration but may not ACQUIT its own Type-B verdict. Pass 1 completeness, Pass 2 DOI resolution, and Pass 3 number matching are Type-A. Pass 2c semantic match and the taste-bearing integrity heuristics are Type-B; route them per `planning-foundation/references/review-loop-protocol.md` rather than self-clearing. ### Fail-loud on missing metadata Verification depends on `metadata.yaml` being complete. If claim-verification encounters `TODO` fields, abort immediately rather than running downstream passes. Verification against incomplete metadata produces a green report the user cannot trust. ### Never auto-edit the manuscript This skill reads `.writing/manuscript/*.tex` and writes `.writing/claims/*.md` STATUS fields and the report. It does not touch manuscript prose. Manuscript changes go through drafting (or the human author's manual edits), which follow the claim-first drafting discipline. ### Report is the audit trail The verify-report.md is the document-of-record for this verification round. It is the evidence that backs the skeleton ("all claims passed citation verification on "). Keep one report per round; when a new round starts, archive old reports under `.writing/archive/reports/`. ## Upstream Skill Contracts This skill invokes these plugin-local skills via the Skill tool with the `superpower-writing:` prefix: | Skill | Invocation point | Expected I/O | |-------|------------------|--------------| | `zotero-mcp` (MCP) | Pass 2a, 2c body lookup, §5 push-back | `zotero_search_items` + `zotero_get_item_metadata` for query-by-DOI; `zotero_semantic_search` for claim-text similarity fallback (catches DOI-mismatched items and finds paragraph-level support when the abstract is ambiguous); `zotero_get_item_fulltext` for narrow passage reads when chunks alone are insufficient; `zotero_add_by_doi` for dedup-aware push. Registered in `.mcp.json`. | | `superpower-writing:citations` | Pass 2b primary | Resolve DOI → canonical Crossref record. | | `superpower-writing:literature` | Pass 2b fallback / semantic match | DOI → abstract; optionally compare abstract ↔ claim text. | A missing skill at invocation time is an unrecoverable error: halt verification and surface the install command. ## Integration Points - **drafting skill:** it flips `stub → evidence_ready` as it resolves evidence. This skill flips `evidence_ready → verified`.