Paper Extract

Paper extract (hybrid)

Do not treat direct PDF reading as the main path. Normalize machine output in _raw/ into extracted/.

Layout

sources/<source-id>/
├── original/          # PDF original (placed by human)
├── _raw/              # Machine extract (not the citation source of truth)
│   ├── marker.md      # Marker full-text draft (single full-text extractor)
│   ├── arxiv/         # arXiv LaTeX / e-print source (authoritative math; preferred)
│   ├── mathpix/       # high-accuracy math pages (fallback for math)
│   └── pages/         # page images pXX.png (optional, for re-extract)
└── extracted/         # human-readable source of truth (cite here)

Path priority (full text)

1. Markdown exists under _raw/ (Marker) → read it and generate/update extracted/
2. _raw/ missing → ask the user to run Marker (see tooling). Do not silently fill extracted/ by reading the whole PDF
3. Exceptions: short checks (e.g., Abstract) or filling pages missing from _raw may use the PDF / page images

Math / equation reference priority

Equations and algorithms must come from an authoritative source, not from full-text OCR. Use this fixed order:

1. arXiv LaTeX / e-print source (_raw/arxiv/) — preferred (compiled TeX; no OCR guesswork)
2. Mathpix (or equivalent high-accuracy math OCR) on the PDF math pages (_raw/mathpix/)
3. PDF page image + vision re-extract of only the affected pages (_raw/pages/)

Fallback on fetch failure (must be explicit)

• If the arXiv source cannot be obtained (paper not on arXiv, source withheld / PDF-only, or the fetch fails), fall to the next path in the list above (Mathpix → page image + vision).
• Do not substitute an undefined or arbitrary source (blog, mirror, a different paper, or model memory) for the missing equations.
• If no path resolves an equation, keep it and mark [extraction uncertain], and state which source was used. When math correctness is load-bearing (survey / both), ask the user before proceeding.

Tooling (Windows-friendly)

Marker: known weaknesses and how to complement them

Marker is the single full-text extractor (Docling is no longer used):

• Equations / inline math can be imperfect → take math from the arXiv source (or Mathpix); do not trust Marker's math for algorithm.md
• Complex / nested tables can be misparsed → keep the caption only, verify against the original PDF page, and note "See original for table"
• Multi-column layout, footnotes, references can be reordered → sanity-check reading order against the PDF when a section looks scrambled
• Figures are captured as captions only (by design) → note "See original for figure"
• Mark any doubtful span [extraction uncertain] and re-extract only those pages via the math reference priority

OCR (Llama) configuration

• Keep the OCR (Llama) configuration — API key, endpoint / model name, and cost-bearing options — under the user's own management (a local, gitignored env / config file that the user owns)
• Reason: it carries credentials and billable settings, so it must never be committed or auto-edited by the agent
• The agent may ask the user to run Llama OCR, but must not generate, hardcode, or commit these settings

By mode:

• study: full-text _raw from Marker is enough to ingest
• survey / both: use the math reference priority (arXiv first) for method / math / algorithm pages; re-extract page-by-page if still broken

Normalize _raw → extracted/

• Do not change meaning. No translation, no “helpful” paraphrase that alters sense
• Split by chapter/section into extracted/NN-<section-slug>.md (~300–500 lines)
• Figures/tables: captions only; "See original for figure"
• Symbol corruption (e.g., π→⇡, ρ→⇢), broken equation line breaks, or likely omissions: mark [extraction uncertain] and, if possible, re-extract only those pages via the math reference priority
• Never cite _raw/ (wikilinks must target extracted/)

Block IDs (Obsidian)

• One ID names one block. Latin letters, digits, hyphens only. Unique within the file. Same-page blocks: ^p12-a, ^p12-b
• Simple paragraph: space + ^id at the end of the last line

Body text of the paragraph. ^p12-a

• Structured blocks (blockquote, list, callout, table): ID on its own line, with a blank line before and after

> Quoted body from the paper.

^p12

Next paragraph.

• Do not invent a range ID to cover several blocks. If later citations need each paragraph, give each its own ID

Roles

Forbidden

• Skipping _raw/ and filling extracted/ by reading the whole PDF
• Treating broken extract as “correct math” without checking
• Using naive OCR (e.g., Tesseract) as the main path for text-layer papers
• On arXiv-source fetch failure, silently pulling equations from an undefined / arbitrary source instead of the defined fallback
• Committing or auto-editing OCR (Llama) credentials / config

Done when

• extracted/ chunks exist; meaning was not changed
• Each extracted/ file has unique block IDs; placement matches the paragraph vs structured-block rules above
• Uncertain math is marked [extraction uncertain] rather than silently “fixed”
• Citations, if any were added in this pass, resolve to existing block IDs