Literature Wiki

Literature Wiki Maintenance (Schema v2)

This file is the canonical operational protocol. Harness-neutral. The skill ships parallel SKILL.md (Claude Code) and AGENTS.md (Codex CLI) bodies kept in sync by hand. If you edit one, edit the other.

Schema v2 (current) introduces controlled vocabulary on the source frontmatter and an update-index operation that augments each concept page (methods/.md, systems/.md, etc.) with an auto-generated "Papers using this" appendix. Pre-v2 wikis can run migrate to backfill.

v2.1 addition: every source page opens with a ## For future agents gist (see "Source page gist" below). Pre-v2.1 source pages get backfilled on next touch or flagged by lint.

Why this skill exists

Most LLM-document workflows look like RAG: every question re-discovers knowledge from raw PDFs. Nothing accumulates. For a researcher reading dozens of papers a year on related topics, this is wasted compounding.

This skill maintains a persistent, structured wiki of markdown files. When the user reads a new paper, the skill reads it, extracts what matters, and integrates it into existing entity / method / system pages — updating cross-references, flagging where new data contradicts old claims, strengthening the evolving synthesis. The wiki is the compiled artifact; the raw PDFs sit beside it but are not the primary lookup layer.

The user curates and asks. The skill writes and maintains.

The pattern is field-independent. Default category names suit physical-science research; users in other fields rename in their <wiki>/CLAUDE.md.

Wiki location

Resolution order:

1. LITERATURE_WIKI_PATH env var.
2. ~/.literature-wiki-path (single-line absolute path).
3. Default: ~/research-wiki/.

If absent, ask before creating.

Wiki layout

<wiki>/
├── CLAUDE.md / AGENTS.md      # ⭐ vocabulary + per-wiki style overrides (v2)
├── index.md                   # navigation hub (manual)
├── log.md                     # chronological log of ingests/queries/lints
├── raw/                       # source PDFs/markdown, IMMUTABLE
│   └── assets/                # downloaded figures
├── sources/                   # one page per paper, summary + key numbers + tags
├── entities/                  # author / group / experiment / code-package pages (manual + AUTO appendix)
├── methods/                   # method-level pages (manual + AUTO appendix)
├── systems/                   # system-level pages (manual + AUTO appendix)
├── observables/               # quantity-level pages (manual + AUTO appendix)
├── debates/                   # explicit pages for who-disagrees-with-whom
└── synthesis/                 # cross-cutting themes, working theses

In v2, the concept pages (entities/<slug>.md, methods/<slug>.md, systems/<slug>.md, observables/<slug>.md) all have the structure:

# <Canonical name from vocabulary>

(manual prose: definition, key references, ongoing debates — written by you and the skill on ingest)

<!-- AUTO:BEGIN -->
## Papers using this <axis> (N)
- [YYYY] Author et al. — one-line claim. → [[../sources/<paper-id>]]

## Cross-tags
methods=[…], systems=[…], observables=[…]
<!-- AUTO:END -->

The manual section is your encyclopedic content. The AUTO block is regenerated by update-index from source page frontmatter.

Controlled vocabulary (the v2 contract)

All tag values come from a controlled vocabulary in <wiki>/CLAUDE.md. Four axes for literature-wiki:

Note: this is NARROWER than the research-profile vocabulary, which has six axes. topics, codes, and collaborators are user-specific concepts that belong in your personal portfolio, not the field-level wiki. However, when you also run research-profile, the slugs for methods, systems, observables should match across both wikis — see "Cross-skill hand-off" below.

Vocabulary file format

In <wiki>/CLAUDE.md:

vocabulary:
  methods:
    cdcc:
      canonical: "Continuum-Discretized Coupled Channels"
      aliases: [CDCC, "continuum-discretized"]
    iav-cdcc:
      canonical: "Ichimura-Austern-Vincent CDCC"
      aliases: [IAV-CDCC]
      parent: cdcc
  entities:
    moro-group:
      canonical: "A.M. Moro group, Universidad de Sevilla"
      aliases: ["Moro group", "Sevilla group"]
  ...

canonical is required; aliases are case-insensitive multi-language; parent propagates to indices.

Vocabulary discipline

When ingesting:

1. Look up each candidate tag against vocabulary (case-insensitive, alias-aware).
2. Known: use canonical slug.
3. Unknown: STOP. Propose vocab diff with definition + aliases. Apply only after explicit user approval.

Never invent a tag silently. Reserved uncategorized slug exists per axis; lint warns if usage > 30%.

Source page frontmatter (v2)

---
type: source
created: YYYY-MM-DD
last_updated: YYYY-MM-DD
title: "Paper title"
authors: [list, lead first]
year: YYYY
venue: "..."
doi: 10.xxx/xxx
arxiv: ...

# Tag axes (controlled vocabulary)
methods: [iav-cdcc, cdcc]
systems: ["d+93Nb"]
observables: [breakup-cs]
entities: [moro-group]

# Cross-references
contradicts: [[sources/<other-paper>]]
extends: [[sources/<other-paper>]]

# Cross-link into research-profile (for the user's own work derived from / responding to this source)
research_profile_links: [[../research-wiki-personal/papers/published/<slug>]]
---

Concept pages (methods/, systems/, etc.) carry simpler frontmatter:

---
type: method | system | observable | entity
slug: <vocabulary slug>
created: YYYY-MM-DD
last_updated: YYYY-MM-DD
last_indexed: YYYY-MM-DD   # set by update-index
---

Source page gist (v2.1)

Immediately after the frontmatter, every sources/*.md opens with a machine preamble:

## For future agents

<2-3 sentences: what the paper claims (with the one key number), which wiki tags /
research lines it touches, and when this page would matter for a query.>

Written so a future agent can decide relevance in one read without loading the rest of the page. Rules:

• Self-contained: no "see below", no reliance on backlinks; name the method/system slugs explicitly.
• Concrete: carry the single most important number or result, not "studies breakup observables".
• This is the triage layer for query; if the gist would not let you decide drill-vs-skip in 10 seconds, rewrite it.

(Pattern borrowed from the AI-first vault rule in obsidian-second-brain; same lineage as Karpathy's LLM wiki.)

Three core operations + two new in v2

1. Ingest

Trigger: user drops a paper into <wiki>/raw/, points at a PDF (or a folder of PDFs) anywhere on disk, or hands DOI/arXiv/URL and says "归档" / "ingest" / "把这篇加进去".

Set consolidation depth first. Not every paper earns the full flow, and the compounding payoff concentrates on papers central to the user's active research lines. Before filing, judge centrality (cross-check index.md, and research-profile if available), then pick the pass:

• Peripheral (off the user's lines, likely cited once): light pass = steps 1, 2, 4 with a lean source page, 6, 8. Skip the deep cross-linking and concept-page rewrites.
• Core (on an active line, likely to be built on or cited): full pass = all steps, with the real work in step 5 (rewrite the affected concept-page prose, do not just append) and in hunting contradictions at step 3.

Scale effort with centrality, not uniformly across every ingest.

Flow:

1. Read the source. If only an identifier given: for an arXiv nucl-th paper, FIRST check the local corpus via a real Skill(skill="literature-corpus", args="show <arxiv_id>") call — if it hits, the paper's full .tex source is already on disk (the returned tex_dir, on the KINGSTON drive), which is character-exact and needs no re-fetch. Use that source. On a miss, or for non-nucl-th / non-arXiv papers, fetch via the user's literature-search skill (anti-hallucination still applies). If neither skill is available, fetch via WebFetch. (The corpus is the wiki's local feed-in; it holds full text but NOT verified BibTeX, so citations still go through literature-search.)
   • PDF-only papers (no .tex / arXiv source) where equations or tables matter: do not read the raw PDF text layer directly for the key-numbers / key-claims extraction; its inline equations come out garbled. FIRST issue a real Skill(skill="pdf-extract", args="<pdf path>") call this turn (a real Skill tool call, not a prose mention; naming it without calling it is the failure mode this guards against), read the returned markdown as the text scaffold, then verify every load-bearing number / equation you record against the original PDF. The PDF is still the file archived into raw/ in step 4; the marker markdown is a working scaffold, not the archived artifact. If pdf-extract is unavailable, read the PDF directly and flag in the source page that equation/table values are unverified. (When a .tex / arXiv source exists, skip this and read the source, which is character-exact.)
2. Discuss with the user. Surface 3–5 key takeaways and ask which to emphasize before filing. Non-skippable on first ingest of a session.
3. Vocabulary check. For each method / system / observable / entity the paper invokes: look up against vocabulary. Unknown → propose vocab diff first, wait for approval.
4. Archive the PDF into raw/, then create sources/<paper-id>.md. FIRST copy the source PDF into <wiki>/raw/ if it is not already there (preserve a stable filename). This is mandatory and applies to EVERY pass including light-bulk: whenever the paper was handed from a path outside the wiki, fetched, or bulk-ingested from an external folder, the file must be physically copied in, not merely referenced. Never leave source_pdf: pointing at a file that is not in raw/. THEN write the page with frontmatter (all v2 fields including tag axes), the ## For future agents gist as the first body section (v2.1, mandatory on every pass including light), bibliographic header, abstract in user's words, key claims with section refs, key numbers, figures of interest (downloaded into raw/assets/), wikilinks to relevant concept pages; set source_pdf: to the archived filename now present in raw/.
5. Update or create concept pages for each tag value used. The manual prose section is updated (new findings, new context); the AUTO block is regenerated by step 7.
6. Update index.md — add the new source under the right category with a one-line summary.
7. Trigger update-index to regenerate AUTO blocks on all touched concept pages.
8. Append to log.md: ## [YYYY-MM-DD] ingest | <paper-id> | <one-line takeaway>.
9. Confirm to the user which pages changed and where contradictions surfaced. A single paper typically touches 5–15 pages (Karpathy's gist reports 10–15 for general articles; scientific papers tend tighter).

Hard invariant (PDF archival). Every sources/*.md page that declares a source_pdf: MUST have that file present in <wiki>/raw/. The wiki is the self-contained archive; a page that names a PDF living only in some external folder (EndNote, Downloads, Zotero) is a broken ingest. This bites hardest on bulk ingests via parallel workers: each worker must cp its PDF into raw/ as part of writing the page, and the orchestrator must verify raw/ file-count rose by the number of new pages before reporting success. After any bulk run, audit with: every source_pdf: value resolves to a real file in raw/ (0 misses), else copy the stragglers in.

2. Query

Trigger: user asks a question wanting wiki synthesis rather than fresh literature search.

Flow:

1. Read index.md to find candidate concept pages by axis (methods, systems, observables, entities).
2. Triage source-page candidates by their ## For future agents gist (first body section, cheap to read); drill into the 3–10 pages whose gist matches, with file:line citations.
3. Synthesize. Flag what is supported, what is contradicted, where the wiki has gaps.
4. If non-trivial, offer to file the answer as a new page in synthesis/ or debates/. Good answers should compound.

3. Lint

Trigger: explicit, or after every ~10 ingests, suggest one.

Check for:

• Contradictions between source pages — flag with [CONTRADICTS sources/X] and create a debates/ entry.
• Stale claims — pages citing old sources where newer ones supersede.
• Orphan source pages — sources tagged with no methods / systems / observables (likely incomplete ingest).
• Missing concept pages — tags used in source frontmatter but no concept page exists.
• Frontmatter drift — wrong type tags, missing dates, off-vocab tags.
• Missing gist — source pages without a ## For future agents first section (pre-v2.1 ingests). Backfill from the page's own content; no PDF re-read needed.
• AUTO-block integrity — concept pages with hand-edited content inside AUTO markers.
• Cross-link validity — contradicts: / extends: / research_profile_links: pointing to non-existent pages.

Output: a markdown report. Apply fixes only after user reviews.

4. update-index (NEW in v2)

Auto-triggered after every ingest; after lint it runs only once the user has approved the lint fixes (lint itself stays read-only). Always idempotent.

Flow:

1. Walk sources/*.md, read frontmatter only.
2. Build axis → tag → sources map. Apply parent: propagation.
3. For each concept page (<axis>/<slug>.md) referenced by any source frontmatter:
   • Ensure the page exists; create stub from vocabulary if not.
   • Locate the <!-- AUTO:BEGIN --> / <!-- AUTO:END --> markers.
   • Regenerate the AUTO block with the current source list:

     <!-- AUTO:BEGIN -->
     ## Papers using this <axis> (N)
     - [YYYY] First-author et al., Venue — one-sentence claim. → [[../sources/<paper-id>]]
     
     ## Cross-tags
     methods: […], systems: […], observables: […]    ← which other tags co-occur most with this one
     
     ## Last indexed: YYYY-MM-DD
     <!-- AUTO:END -->
     

4. For concept pages whose tag has been removed from all sources, mark as ## Status: orphaned in the manual section header (do not delete; the prose may still have value).
5. Report changes.

5. migrate (NEW in v2)

For wikis built before v2 (no controlled vocabulary, no AUTO blocks on concept pages):

1. Walk sources/*.md. For each, identify missing v2 fields (tag axes, cross-references). Propose values from existing wikilinks; user confirms / corrects.
2. Walk concept pages. For each, insert <!-- AUTO:BEGIN/END --> markers if absent (place at the bottom). Existing prose is preserved as the manual section above the markers.
3. Run update-index once at the end to populate AUTO blocks.
4. Append a migration log entry.

Interactive and slow on a large wiki — that is intentional. User must validate each tag application; silent migration would defeat controlled vocabulary.

Working principles

• The user curates, the skill writes. Pages contain only content from real sources or the user's explicit input.
• Quote with provenance. Every non-trivial claim links back to a source page or external citation, never floats free.
• Lean over comprehensive. 80% covered and trusted beats 100% covered and unread.
• Use [[wikilinks]] for internal references. Survives renames; Obsidian renders.
• Read the wiki's CLAUDE.md / AGENTS.md first for vocabulary and personal style overrides.
• Controlled vocabulary is a contract. Never invent tags silently; the cost of asking is small.
• Frontmatter drives indices, body holds narrative.

Cross-skill hand-off

To/from research-profile

The user's own portfolio lives in research-profile (~/research-wiki-personal/). The two wikis cross-link via:

• Shared vocabulary slugs for methods, systems, observables, collaborators (where the latter two skills overlap). Recommended: keep a single ~/research-vocabulary.yml and have both wikis' CLAUDE.md files import from it (skill supports include: <path> in vocabulary block).
• research_profile_links: frontmatter field on source pages — when the user has their own paper that responds to / extends / contradicts this source, link from here to there.
• Reverse direction: research-profile pages add literature: [[../research-wiki/sources/<paper-id>]] when they cite a paper in this wiki.

To/from literature-search

A literature-search skill (database fetcher with anti-hallucination) is the recommended source of new papers. Output of literature-search is input to this skill via <wiki>/raw/. After a successful literature-search, suggest filing the new sources here if the user maintains a wiki.

To/from literature-corpus

A literature-corpus skill holds the full local arXiv nucl-th corpus (~62k papers, full text, lexical/BM25). It is this wiki's local feed-in: on ingest of a nucl-th arXiv paper, Skill(skill="literature-corpus", args="show <arxiv_id>") locates the character-exact .tex source already on disk (step 1 above), avoiding a re-fetch. It is also the discovery layer for "what in the whole field bears on this topic" beyond the few-hundred papers already in the wiki — a corpus hit worth keeping becomes an ingest candidate here. The corpus has full text + metadata but NOT verified BibTeX, so any citation still passes through literature-search. A corpus miss is never proof of novelty (nucl-th-only, lexical).

To/from paper-writing skills

Any paper-writing skill (review-writing, journal-submission) should query this wiki when building related-work sections. The query operation here returns [[wikilinks]] that paper-writing can resolve to BibTeX (via literature-search verification).

Templates shipped

• templates/CLAUDE.md — vocabulary starter with empty axes (entities/methods/systems/observables) + example entries. Copy to <wiki>/CLAUDE.md on first use.