wuyoscar arxiv2agent — structured paper reading for agents
Setup (once)
Check availability first; install only if missing:
arxiv2agent --help || (git clone https://github.com/wuyoscar/arxiv2agent /tmp/arxiv2agent \
&& cd /tmp/arxiv2agent && uv tool install .)
Digest papers
arxiv2agent 2305.13860 -o corpus/ # one paper
arxiv2agent 2305.13860 1706.03762 2005.14165 -o corpus/ # many: pass a LIST of IDs
- For multiple papers ALWAYS pass them as a list to ONE command (sequential + rate-limited internally; one failure doesn't abort the batch). Do NOT run one command per paper, and do NOT parallelize.
- Cheap to re-run: sources and metadata are cached locally; a cached paper digests in ~0.2s offline. 10 new papers ≈ 40s end to end.
- No arXiv ID? Resolve title → ID first (one web search:
site:arxiv.org <title>), then digest. Never answer content questions from search snippets.
Read the digest — rules
The digest at corpus/<id>/ is the full paper. Read files, don't search:
| you need | read |
|---|---|
| overview / navigation | README.md (outline + entity index with stable IDs) |
| one section | sections/NN-slug.md (YAML frontmatter + markdown) |
| everything, in bulk | paper.json — fixed schema across ALL papers |
| a figure | figures/fig-<slug>.json + its image files; text-body figures (prompt boxes, examples) are in figures/fig-<slug>.txt |
| a table / equation / algo | tables/tab-*.tex, equations/eq-*.tex, algorithms/alg-*.tex |
| code from the paper | listings/lst-*.py (real, runnable extensions) |
what [@key] cites |
references.json |
footnote [^fn:N] |
footnotes.json |
Inline markers in section text: [@key] = citation, [#fig:x]/[#tab:x]/[#eq:x] = entity reference (the ID after # matches the entity's id in paper.json exactly).
Completeness rule: when assessing a paper's claims, do not stop at the Abstract — the sections are all local and cheap. Check is_appendix (on sections AND entities) to distinguish main-text evidence from appendix evidence.
Honesty fields: metadata.*_source says how each field was extracted (arxiv_api, title_cmd, none…). A citation with title: null means the paper shipped no .bib — quote bib_raw/key, don't guess the reference. warnings.residue_top lists LaTeX that survived cleaning; mention it if quoting affected sections.
Program-calling patterns (preferred for ≥2 papers)
Don't cat paper-by-paper. Write one small script over paper.json:
import json
from pathlib import Path
def load(pid, root="corpus"):
return json.loads(Path(root, pid, "paper.json").read_text())
# Compare the Method sections of many papers
ids = ["2305.13860", "1706.03762", "2005.14165"]
methods = {
pid: "\n\n".join(s["text"] for s in load(pid)["sections"]
if "method" in s["title"].lower() and not s["is_appendix"])
for pid in ids
}
# Which sections cite a given work?
cited_in = {c["key"]: c["cited_in"] for c in load("2305.13860")["citations"]}
# All prompt/example text-figures of an LLM paper (main text only)
prompts = [f["text"] for f in load("2005.14165")["figures"]
if f["body_tex"] and not f["is_appendix"]]
The schema is identical for every paper (schema_version in paper.json), so these loops never need per-paper special-casing.