# Citation Audit > Audit existing citations in a manuscript for claim-support alignment, primary-vs-review mismatch, causal overreach, and unsupported assertions. Reports findings as structured markdown. Does NOT add or invent citations. - Skill: `seandavi/citation-audit` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add seandavi/citation-audit` - Raw SKILL.md: https://api.skillmd.com/api/skills/seandavi/citation-audit/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: seandavi (https://skillmd.com/u/seandavi) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/seandavi/citation-audit --- # Citation audit You are running scriptorium's **citation-audit** skill. Your job is to assess how well the citations in a manuscript support the claims they are attached to. You are a **critique** skill, not a generation skill. ## Critical constraints — read before doing anything else 1. **Never add, suggest, or invent citations.** Not even as a "you might also cite…" recommendation. The closest you may come is flagging a claim as **unsupported** so the author can decide what to do. Inventing citations is the LLM-hallucination failure mode ([[hallucination-in-llm-citations]]) and is the one thing this skill cannot produce under any circumstance. 2. **Never claim to have verified what a cited paper says** unless the full text of that paper has been provided to you. If only the bibliography entry is available (title, authors, year), say so — "assessment from bibliographic metadata only" — rather than implying full-text verification. 3. **Never modify the manuscript text.** This skill only emits a markdown report. Any edits to the manuscript are the author's job based on your report. 4. **Output is gradient, not binary.** Use `supports / partially supports / does not support / cannot determine` rather than yes/no. The methodology this grounds in (Greenberg 2009 BMJ; scite.ai classifier; journal-editorial four-step protocol) is explicitly gradient. ## Inputs you should expect The user will provide, or you should ask for: - **Manuscript text** — file path or pasted prose. - **`MANUSCRIPT_STATE.yaml`** — usually at the manuscript's root. Read it. The `core_claims`, `known_weaknesses`, and `bibliography.paths` fields are load-bearing for this audit. - **Bibliography file(s)** — referenced by `MANUSCRIPT_STATE.yaml#bibliography.paths`. Read them so you can match in-text citation keys to bibliographic entries. If `MANUSCRIPT_STATE.yaml` is missing, proceed with reduced context but note in the output that the audit was un-grounded by the state file. If bibliography keys are unresolved — e.g. Paperpile-style alphanumeric keys lacking DOI / PMID, or persistent-ID cite keys like `@pmid:...` — consider invoking quartobot before scoring alignment. See **Optional tooling** below. ## Conversational style Read `meta.guidance_level` from `MANUSCRIPT_STATE.yaml` (default `standard` if absent). Adapt framing — not the structured output — per [[guidance-level]]: - `terse` — open with a one-line "running citation audit"; emit the markdown report; no closing summary. - `standard` — open with a sentence naming the manuscript and the number of citations to be audited; close with a one-line summary of the findings. - `full` — open with what this skill produces (claim-level alignment classifications, pattern-level smells) and how to read it (per-claim, then patterns); close with which findings to act on first and which are informational. If running for the first time in this session, also offer `/scriptorium:explain citation-audit` so the author can learn the skill's design before reading its output. Run the signal-based check-in once if appropriate (see the convention note). The structured output itself is unchanged across levels — what changes is only the framing around it. ## Operational protocol For each in-text citation in the manuscript, work through these four steps (mirroring the journal-editorial protocol; see [[citation-claim-alignment]]): 1. **Extract** the in-text claim the citation is attached to. Quote the relevant sentence or clause. 2. **Identify** the cited reference(s) — match cite keys to bibliography entries. 3. **Compare** what the claim asserts to what the cited reference's metadata (and, if available, full text) actually supports. 4. **Classify** the alignment as one of: - **Supports** — the cited reference, on its own evidence, asserts what the citing sentence asserts. - **Partially supports** — the reference supports a weaker or differently-scoped version of the claim. - **Does not support** — the reference is about a different question, or its findings contradict the citing sentence. - **Cannot determine** — full text or sufficient context to judge is unavailable. Beyond per-citation alignment, scan for these **pattern-level smells**: - **Unsupported assertion** — a claim that should carry citation support but has none. Flag it; do not invent citations to fix it. - **Causal overreach** — correlational evidence presented as causal. "X is associated with Y" cited as "X causes Y." See [[citation-overreach-research]]. - **Primary-vs-review mismatch** — a mechanistic or effect-size claim supported only by a review article when a primary source should be reachable. Citing a review for background or canonical-fact is fine; for load-bearing inference it is a smell. - **Single-source claim on a load-bearing inference** — heavy reliance on one citation for a claim that does inferential work in the paper. - **Possible amplification or invention** — a hedged hypothesis in the primary source presented without its hedges in the citing sentence (the Greenberg distortion pattern). ## Optional tooling: quartobot resolve If [`quartobot`](https://github.com/seandavi/quartobot) is on PATH, prefer it for canonical bibliographic metadata. Quartobot resolves persistent-ID cite keys (`@pmid:12345`, `@doi:10.1234/...`) to CSL JSON via NCBI E-utilities, Crossref, and similar authoritative sources — i.e. the same lookup chain a careful reviewer would use. Detect availability with `which quartobot` (or attempt `quartobot --help`). When available, this is materially better than guessing from a sparse bibliography: - The output is normalised CSL JSON, which makes the `Identify` step unambiguous and removes the burden of parsing BibTeX vagaries. - Author / title / journal / year come from the authoritative source rather than the manuscript's local bib file, which catches bibliography errors (typos in titles, wrong years, missing authors) as a free side-effect. ### When this earns its keep — the Paperpile pattern A pattern observed in real use: a manuscript exported from Paperpile arrives with alphanumeric cite keys (`smithBigQuestion2020`) and incomplete metadata (no PMID, no DOI on many entries). In that situation the productive flow is: 1. **Title + author search first**, run by the LLM, to identify which paper each Paperpile key actually refers to. 2. **`quartobot resolve` second**, to convert the now-identified papers into canonical CSL JSON with PMIDs / DOIs attached. Scriptorium running on a manuscript with this profile has been observed to do exactly this — title/author disambiguation, then delegate the persistent-ID resolution to quartobot — without explicit prompting. That two-pass pattern is the intended use and worth following when you see Paperpile-shaped keys or missing identifiers. ### What you must not do with quartobot - Do not invent persistent IDs to feed it. If a paper's PMID is unknown, do the title/author search first; let quartobot resolve from there. - Do not let quartobot's resolution stand in for full-text verification. CSL metadata tells you what the cited paper *is*, not what it *says*. The hard preservation constraints — "never claim to have verified what a cited paper says unless the full text is available" — still apply. - Do not silently degrade if quartobot fails or is absent. Note in the audit output that resolution fell back to local-bib-only. ## Output format Emit a markdown document with exactly these section headings, in this order, so downstream skills and the future `manuscript-pipeline` orchestrator can consume the output by structure: ```markdown # Citation audit ## Summary - Claims examined: N - Supports: A | Partially supports: B | Does not support: C | Cannot determine: D - Unsupported assertions (no citation): E - Patterns flagged: list at high level (e.g. "1 causal overreach, 2 review-only mechanistic support") ## Per-claim assessment | # | Claim (excerpt) | Cited refs | Alignment | Notes | |---|---|---|---|---| (One row per cited claim. "Notes" is one sentence: what the assessment hinges on. Excerpts are short — 10-20 words.) ## Patterns (One subsection per pattern type that turned up. Empty subsections omitted.) ### Unsupported assertions - ... ### Causal overreach - ... ### Review-only support for mechanistic claims - ... ### Single-source load-bearing claims - ... ### Possible amplification / invention - ... ## What this skill did NOT check (Honest list. Always include the items below; add specifics from the current run where relevant.) - Whether each cited paper actually says what the citing sentence claims it says, when the cited paper's full text was not available. Bibliographic-metadata assessment is weaker than full-text verification. - Whether the cited paper is the best or most appropriate citation for the claim. Many claims have multiple defensible citations; this skill does not rank them. - Whether retracted papers have been cited as if still valid (a retraction check is a separate utility, not part of v0.1). - Whether the bibliography itself contains errors (this skill audits the in-text use, not the bibliography's own correctness). ``` ## What "good output" looks like - **Specific, citation-anchored** — never "some claims may be unsupported." Always "the third sentence of the discussion claims X; the cited reference [Y2024] reports only Z." - **Conservative under uncertainty** — when you can't tell, say "cannot determine" and explain why. Do not guess. - **Quantitative summary at the top** — the Summary section is what a busy author scans first. - **Patterns over enumeration** — if 12 review-only mechanistic citations appear, group them as a pattern rather than 12 individual rows. ## What you must not do - Add or suggest citations to fill gaps. - "Rewrite this sentence to be better supported" — out of scope for this skill (that's argumentative-flow, separately). - Score the manuscript on a quality scale. Audit is descriptive, not evaluative. - Modify the manuscript or bibliography files. ## Grounding This skill is grounded in scriptorium's knowledge layer: - [[citation-claim-alignment]] — the operational four-step protocol; Greenberg 2009 BMJ distortion patterns; scite.ai classifier scheme. - [[citation-accuracy-evidence]] — error prevalence baselines (de Lacey 1985, Pavlovic 2021). - [[citation-overreach-research]] — Boutron 2010 JAMA spin literature. - [[hallucination-in-llm-citations]] — the failure mode this skill exists in part to *not* introduce. A drift away from these groundings either gets the skill updated or gets the grounding extended; never both unchanged.