Paper Ingest

Paper ingest

When a PDF exists under sources/<source-id>/original/, run the full set below. Extraction details: read /paper-extract when math or algorithms are broken, _raw/ is missing or the main task, or Marker / OCR / arXiv math is in play.

Templates: _templates/source-index.md, _templates/summary.md. Also _templates/study-note.md, _templates/question-bank.md, _templates/algorithm.md, _templates/repro-readme.md as mode requires.

Replace every {{...}} in copied templates (source-id, title, mode, date). Do not copy unused placeholder text into real notes.

Steps

1. Confirm source-id and mode (study / survey / both). Ask if unspecified
2. Confirm extraction inputs (/paper-extract)
   • If _raw/ exists, use it as the primary input
   • If missing, ask the user to run Marker (full text); for math, use the reference priority (arXiv source first). Continue only after inputs are ready
   • Do not full-read the PDF to fill extracted/ (short checks and missing-page fills only)
3. Normalize _raw/ into section chunks extracted/NN-<section-slug>.md
   • About 300–500 lines per file
   • Add block IDs per /paper-extract (one ID = one block; unique in the file)
   • Figures/tables: captions only; note "See original for figure"
   • Do not change meaning. Mark doubtful extractions with [extraction uncertain]
   • For survey/both, if math or algorithms are broken, follow /paper-extract (arXiv source → Mathpix → page image + vision); on failure, follow that fallback and mark [extraction uncertain] (never an arbitrary source)
4. Generate index.md from _templates/source-index.md. Set mode from step 1. Keep only Artifact links that match mode
5. Generate summary.md from _templates/summary.md (every claim must be cited per CLAUDE.md). Set the same mode
6. Register the paper in _index/MOC.md and _index/keywords.md. MOC status must match index.md frontmatter. On first paper, delete the (none yet) row
7. If mode is study/both, prepare study/ stubs from _templates/study-note.md and _templates/question-bank.md. If survey/both, prepare survey/algorithm.md from _templates/algorithm.md and survey/repro/README.md from _templates/repro-readme.md
8. Set status to ready on index.md, summary.md, and any stubs created in step 7 (unless extraction is still incomplete: then extracting)

Done when

• extracted/ chunks exist via _raw/ or an explicit exception path
• Block IDs in each extracted/ file are unique
• index.md / summary.md exist; every #^ citation resolves to a block ID that exists in the target file
• No {{...}}, …, or other template wording remains in generated files
• Unused study/survey Artifact links were removed
• Frontmatter mode matches the confirmed mode; status is ready (or extracting if still incomplete); authors / year / venue are filled only from the paper
• The paper is listed in _index/ with the same mode and status
• Citations point only to extracted/ (never _raw/)