Citation Audit

Citation Affiliation Audit

Find citing papers where at least one author is affiliated with a notable institution. Surfaces adoption evidence that web search and PDF search miss entirely.

Important framing: This finds "researchers AT [institution] cited your tool", not "[institution] officially endorses your tool." The distinction matters for grant narratives and tenure materials. Use the framing "researchers at NASA / JPL / NIH cited [paper]" rather than "NASA cited [paper]" unless the citing paper itself names the institution as the work's home.

When to Use

• Before tenure / promotion materials
• Before grant applications requiring broader-impact evidence
• After major citation milestones (1000+ Scholar citations, an inclusion in a foundation-model system card)
• Once per semester as a periodic audit
• As a sub-step inside the [[news-search]] full audit (the news-search skill hooks this audit's output into its final report; see "Integration with news-search" below)

Inputs

• data/publications.json -- all papers (titles, venues, years, paper_url for arXiv/DOI extraction)
• (Optional, for --source both / dimensions) .env with DIMENSIONS_API_KEY from Digital Science scientometric tier
• Prior citation-affiliation-audit.md if it exists (this run overwrites it; older Tier 0 rows can be eyeballed for stability)

Sources

Two bibliometric sources, orchestrated by scripts/citation_affiliation_audit.py:

• OpenAlex (no credentials required, but metered). Standard open citation graph. Undercounts CS papers -- PyOD shows ~24 citations on OpenAlex versus 1,000+ on Google Scholar -- but covers a broad swath of the literature and is the no-credential default. OpenAlex now bills by request. The free tier is 1000 credits (US$0.10) per day, reset at midnight UTC, and every API call costs one credit. A single full audit resolves 116 paper identifiers before it queries a single citing paper, so the budget can be gone before Phase 2 starts. Phase 1 identifier resolutions are therefore cached in .cache/openalex-id-cache.json; a re-run reads from the cache and spends its budget on citing-paper queries instead. When the budget runs out the script aborts the OpenAlex pass, marks the remaining works as never queried, writes a warning block into the report, and exits 3. Read that state as "unknown", never as zero.
• Dimensions Analytics API (requires DIMENSIONS_API_KEY in .env, scientometric researcher access tier from Digital Science). Stronger CS coverage in practice, plus GRID-normalized research_orgs institution affiliations.

Measured 2026-09-12: on a 14-record sample of recent arXiv preprints, the queried services did not recover the edges that direct document reads found. A cross-vendor sweep of the 14 agent-auditing works queried eight citation indexes. OpenAlex resolved all 14 by DOI 10.48550/arxiv.<id> with correct titles and dates, then returned referenced_works_count = 0 for every one and cited_by_api_url = null, while Semantic Scholar served 25 references and 3 citations for the same Praetor record. Its list endpoints were hard-429 for the whole run with Retry-After: 69687s (about 19.4 hours) on two separate egress paths. Two limits on how far that generalizes: an empty outgoing reference list does not establish that nothing cites the target, since OpenAlex models outgoing references and incoming citations separately, and a 429 is not a successful empty list. Fourteen records are a sample rather than the population. The practical consequence for this skill: a Tier 0 or Tier 1 row absent from an OpenAlex-only run is weak evidence of absence for a recent preprint, so run --source both whenever the Dimensions key is available, and confirm any absence that matters with a direct document check.

Two further limits, same measurement. Inciteful is not an independent check on OpenAlex: it returned zero in both directions for all 14 targets and all 18 known citers, reproducing the same blind spot rather than testing it. And a reference that carries only a repository URL has nothing for an identifier-keyed audit to match. Two of the seven citations confirmed that day cite github.com/HeadyZhang/agent-audit with no arXiv identifier in the bibliography entry. Both benchmark the scanner with numbers, as do two further citers that do carry identifiers, so four of the seven report measurements. A DOI-keyed or identifier-keyed audit registers neither repository-only entry, so a repository-URL search leg belongs in any sweep of software-bearing work.

The orchestrator defaults to --source openalex. Use --source both for the strongest combined audit when Dimensions credentials are configured; --source dimensions runs only Dimensions.

Pipeline

1. Load ALL papers from data/publications.json (not a hand-picked subset). Only surveys are excluded via title keywords (comprehensive survey, a survey on, a survey of).
2. Locate each paper on every enabled source by arXiv ID, then DOI, then a strict title match. Title matching uses a two-pass strategy: exact normalized-title match first, then prefix-anchor (pre-colon acronym like PyOD:, TrustGen:) plus high word overlap (50%+ with prefix, 70%+ without).
3. Query citing papers per work. OpenAlex uses filter=cites:{work_id}; Dimensions uses where reference_ids = "<pub_id>". Per-work queries preserve the mapping from each citing paper back to which of your works it cites; aggregate batch queries would lose that attribution.
4. Extract author institution affiliations from every citing paper. Dimensions provides GRID-normalized research_orgs directly; OpenAlex returns authorships[].institutions[].display_name. The shared classifier in scripts/citation_audit_common.py matches both against the same Tier 0 / Tier 1 regex banks (see "Tier definitions" below).
5. Deduplicate by (institution, citing title, cited work). A citation seen by both sources collapses to one row with source = openalex+dimensions.
6. Write results to citation-affiliation-audit.md at the repo root, with a Source column on the Tier 0 / Tier 1 tables and per-source Coverage subsections.

The shared logic lives in scripts/citation_audit_common.py. The two source modules (scripts/citation_affiliation_audit.py for OpenAlex, scripts/citation_affiliation_audit_dimensions.py for Dimensions) can also be invoked standalone for debugging.

Comprehensiveness Rule

Search every paper, not just the high-profile ones. You do not know where the gold lies. The ESA OPS-SAT citation came from PyOD (obvious), but NASA JPL cited a less obvious paper. The CDC cited ECOD, not PyOD. Deutsche Bundesbank cited ECOD. Morgan Stanley cited XGBOD. Coverage can come from any paper.

The script reads data/publications.json and searches ALL non-survey entries. Do not skip papers because they have low citation counts or seem unlikely to have notable adopters.

Tier Definitions

The authoritative source for tier classification is the regex banks in scripts/citation_audit_common.py (TIER0_PATTERNS and TIER1_PATTERNS). The lists below are a human-readable summary of what those banks match as of the latest commit; if a name is in one place but not the other, the code wins and this prose is stale.

The classifier uses regex patterns on institution names:

• Tier 0: Government agencies (NIST, NASA / JPL, NIH, CDC, FDA, DARPA, NOAA, FAA, CISA, DHS), space agencies (ESA, JAXA, ISRO, DLR, CNES), national labs (LANL, LLNL, Sandia, ORNL, Argonne, Brookhaven, PNNL, INL, Fermilab, SLAC), defense contractors (Raytheon, Lockheed, Northrop, BAE, MITRE, RAND, Booz Allen, Leidos), foundation model companies (OpenAI, Anthropic, DeepMind, Meta AI), central banks (Federal Reserve, ECB, BIS, Bundesbank, Bank of England, BoJ, PBoC), and international bodies (WHO, IAEA, World Bank, OECD, NATO).
• Tier 1: Big Tech (Google, Microsoft, Amazon, Apple, NVIDIA, Intel, IBM, Salesforce, Adobe, Samsung, Huawei, Tencent, Alibaba, Baidu, ByteDance), finance (Goldman, JPMorgan, Morgan Stanley, Citadel, Two Sigma, BlackRock, Bloomberg, Capital One, Wells Fargo, Visa), pharma / healthcare (Pfizer, Roche, Novartis, AstraZeneca, Merck, Eli Lilly, Sanofi, Moderna, Mayo Clinic, Cleveland Clinic), industrial (Siemens, Bosch, Honeywell, GE), telecom (Ericsson, Nokia, Cisco, Qualcomm), automotive / aerospace (Tesla, SpaceX), retail (Walmart), consulting (Deloitte, McKinsey, PwC, Accenture).

Tier order matters: a citing-paper row matches at most one tier, Tier 0 wins on tie. Some patterns use \b boundaries on short acronyms (e.g., \bNIST\b) to dodge substring collisions.

When adding a new pattern, place Tier 0 organizations in the Tier 0 list and check for substring conflicts with existing patterns (e.g., Google is in Tier 1 with a (?!.*DeepMind) exclusion so Google DeepMind lands in Tier 0).

What to Exclude

• Survey papers are filtered automatically by title keyword. OpenAlex conflates papers with similar titles, producing high false-positive rates on surveys.
• Papers with ambiguous short names (e.g., BOND matching biology papers). The script uses the same prefix-anchor + word-overlap thresholds described in step 2 of the Pipeline section (50% overlap with a prefix-anchor like PyOD:, 70% without). If the match is rejected, the paper appears in the not_found list.
• Grant-funded affiliations vs. employment: NIH appears as an affiliation on many papers, but most are university researchers with NIH grants, not NIH intramural staff. The output uses the framing "researchers AT [institution]" to avoid overclaiming. NIH Clinical Center and NIH intramural programs are stronger signals than generic National Institutes of Health.

Run Modes

Full combined run takes ~50-80 minutes (OpenAlex pass ~30-60 min + Dimensions pass ~10-20 min). Use --limit N to scope to the first N papers for a quick verification pass.

Known Limitations

• Coverage gaps still exist with both sources. Even with --source both, the audit lags Google Scholar on CS citation counts. Treat the output as a lower-bound estimate, not a census.

• Google Scholar has no API. Scholar has the most complete citation data but cannot be queried programmatically. For high-priority papers where the combined audit returns few results, run a manual Scholar spot-check.

• Semantic Scholar has good citing-paper recall on some venues but affiliation data is mostly empty, so it is weak for this specific audit. Useful only as a third-pass title-discovery step.

• Re-run periodically. Both OpenAlex and Dimensions backfill over time. A paper that shows 0 citations today may show 50 in six months. Re-run at least once per semester. Recent preprints (2026) will only appear in future runs.

• OpenAlex zeros have two different causes; separate them before believing either.

  1. Budget exhaustion (the dominant cause, diagnosed 2026-08-13). OpenAlex returns HTTP 429 with Retry-After measured in hours once the daily credit budget is spent. Before the 2026-08-13 fix the script retried five times, gave up, returned an empty list, and printed 0 citing papers -- so an API failure was recorded as a zero-citation finding. That is the real mechanism behind the OpenAlex: 45 papers with citations; 0 unique citing papers analyzed line in the 2026-08-09 report, and behind every earlier report carrying the same shape. Phase 1 selects works because they have citations. Phase 2 then returning zero for all of them is arithmetically impossible, so that line was always a failure signature. The script now aborts on a long Retry-After and says so. If a report carries the warning block, the OpenAlex numbers are a floor and nothing more.
  2. Genuine indexing lag on recent CS preprints. Real, and separate from the above. The 2026-07-19 round found Auditable Agents (arXiv:2604.05485) and Implicit Execution Tracing (arXiv:2603.17445) at OpenAlex 0 against seven and several verified citations. For any paper under about 12 months old, cross-check with the Semantic Scholar citations API (api.semanticscholar.org/graph/v1/paper/arXiv:<id>/citations) before recording a zero.

  Both causes point the same way: prefer --source both when Dimensions credentials exist, and never let an OpenAlex zero silently zero out a paper in the audit tables.

• Identifier resolution order. Each source tries arXiv ID, then DOI, then a strict title match. arXiv lookup is the most reliable for CS papers but still requires the source to have ingested the paper.

• Source disagreements are signal. When OpenAlex and Dimensions disagree on whether a paper has notable citations, prefer the higher-recall source for follow-up rather than picking one as ground truth.

• Exact-string cross-source dedup. The merge key is (institution, citing_title, cited_work). Variants like Google vs Google LLC or punctuation-variant titles may produce near-duplicate rows that span sources. The audit Markdown footer carries a caveat. Fuzzy dedup is deferred until the audit becomes the source of record for exact counts.

Improving Coverage

OpenAlex alone covers ~50% of papers as of Apr 2026. The integrated Dimensions Analytics API path closes a meaningful share of that gap on CS papers; run --source both for the strongest combined coverage.

Setup (one-time):

1. Get a key from app.dimensions.ai -> account settings -> API (scientometric researcher access tier from Digital Science).
2. Add DIMENSIONS_API_KEY=<key> to .env (see .env.example). Access expires 2027-01-01 unless renewed.
3. pip install dimcli python-dotenv if not already present.
4. Verify with python skills/citation-audit/scripts/dimensions_smoke_test.py before the first real run; the smoke test checks research_orgs, citation graph fields, and classification access.

Additional gap-closers still useful when Dimensions falls short:

• Semantic Scholar API (api.semanticscholar.org/graph/v1/paper/{id}/citations): better CS coverage than OpenAlex on some venues, but affiliation data is mostly empty. Useful as a third-pass title-discovery step, then cross-reference titles back to OpenAlex or Dimensions for institution data.
• Web of Science / Scopus APIs: best affiliation data but require USC library credentials. Use for manual verification of Tier 0 claims that show up in only one source.
• Re-running the script every 3-6 months is the simplest way to improve coverage, since both OpenAlex and Dimensions backfill continuously.

Output

Results go to citation-affiliation-audit.md at the repo root. The file includes:

• Header: per-source headline numbers (papers with citations + unique citing papers per source), so combined runs do not present max(...) as a single combined truth.
• Tier 0 table: Category, Institution, Country, Your Work Cited, Citing Paper, Year, Source.
• Tier 1 table: same columns.
• Summary by institution: aggregated counts per (institution, category).
• Coverage subsections: per-source breakdown of papers with citations, indexed-but-zero-citations, and not-found lists. Per-source so a paper missing from OpenAlex but found by Dimensions does not appear in a combined "not found" list.
• Methodology footer: framing reminder (researchers AT institution, not endorsement) + cross-source dedup caveat.

Integration with [[news-search]]

The [[news-search]] skill runs editorial-coverage discovery across D1-D8 + D10 (press, government PDFs, ecosystem, deep-research-tool output). Its final report is news-coverage-audit.md. When the user runs the news-search Full audit, it should read the existing citation-affiliation-audit.md and integrate the full citation evidence as a cross-skill section. This contract is authored here and mirrored semantically in skills/news-search/SKILL.md's "Cross-skill: citation-audit integration" section (same three states, same full-table embedding rule, same no-silent-auto-invoke policy; wording differs to fit each skill's surrounding context). If you change one side, update the other in the same commit.

1. Before starting news-search Full audit, check whether citation-affiliation-audit.md exists at the project root and record its modification time.
2. If the file is missing, tell the user: "Citation affiliation audit has never run on this project; recommend running /citation-audit --source both (or --source openalex if no Dimensions credentials) before the news-search full audit." Do not auto-invoke without confirmation; a full citation audit is 30-80 minutes and may use Dimensions quota.
3. If the file is fresh (modified within the last 30 days), copy the citation audit's full Tier 0 table, full Tier 1 table, Summary by Institution subsection, and per-source Coverage subsections verbatim into news-coverage-audit.md under ## Citation Affiliation Evidence (integrated from citation-audit). Add a one-line freshness stamp and a link back to citation-affiliation-audit.md as the canonical separate copy.
4. If the file is stale (older than 30 days), integrate the same full content, mark the section header as ## Citation Affiliation Evidence (stale, regenerate via /citation-audit), and surface the staleness to the user. Do not silently truncate to a top-hit summary.

The hook is read-only from news-search's perspective. It never re-runs the citation audit silently; the user decides whether to regenerate. Verbatim full-table embedding is intentional: news-coverage-audit.md is the single artifact tenure / promotion or grant reviewers scan, so the impact story must be self-contained rather than asking the reader to follow a link.

Related Skills

• [[news-search]]: editorial coverage of FORTIS work (press, government PDFs, ecosystem, deep-research-tool output). Pairs with this skill for combined external-impact evidence.
• [[bibref-verify]]: audits an existing .bib for hallucinated entries before paper submission. Not related to citation-affiliation audit, but adjacent in bibliometric workflows.