Update References

Update references (research paper search)

This skill turns "find me what's been written about this" into a small, repeatable pipeline: read what the user is working on, search arXiv (and, for off-arXiv work, Semantic Scholar or Google Scholar) for related papers, download the relevant ones, read them, write a summary per paper, and keep a Citation Style Language (CSL) YAML database so work is never repeated and versions stay current.

The standing interests are LLMs, AI agents, finite state machines, and declarative agent patterns — the territory of spindle (github.com/petar-djukic/spindle), a Go state-machine engine for agentic loops. Lean toward those topics, but always sharpen the search using whatever the current working directory is actually about.

Where things live

This skill is context-aware. It runs from the directory the user is working in, and it stores its outputs there — not in a fixed global location. Resolve paths like this:

• Database: references.yaml under the current working directory, unless an existing one is already nearby (look for a references.yaml at or above the working directory and reuse it). Pass the chosen path as --db to every script call so search, fetch, and record all agree. If only a legacy arxiv/papers.yaml exists, the scripts read it transparently and convert on next write.
• PDFs: <db-dir>/pdfs/ (transient — they exist to be read; they don't need to be committed). The script resolves this relative to the database, not the current directory, so PDFs land next to the db even when run from elsewhere.
• Papers (markdown): <db-dir>/papers/ — fetch converts each PDF to markdown automatically (best-effort via pymupdf4llm), preserving headings, tables, and math. Papers are readable in any editor or tool.
• Summaries: <db-dir>/summaries/, one file per paper.

The PDF, markdown, and summary of a paper share one human-friendly stem: <Family>-<Year>-<title-slug>-<source>-<id> (for example Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052). The <source>-<id> tail is arxiv-<id>vN, doi-<doi>, or scholar-<key>, so any file traces back to its source at a glance. This differs from the pandoc citation key, which stays short — just <family>-<year> (see the database format). repair renames files created under the older naming scheme.

• Manual-download list: <db-dir>/downloads-needed.md — regenerated by scholar.py pending, a checklist of papers that could not be downloaded (paywalled or behind a login) with a landing URL for each. Derived from the database, so it is disposable, not hand-maintained. An entry carrying download: deferred is kept off the checklist (it lists compactly at the bottom instead) — set it on metadata-only entries the current work does not cite, and remove it when a citation pass starts citing the entry.

If the user names a directory or an existing database, use that instead. Because every output path is derived from --db, passing an absolute --db keeps all artifacts together regardless of the working directory.

The database format

The database is CSL (Citation Style Language) YAML — by default a bare list with no root key. Each entry has standard CSL fields that pandoc understands (id, type, title, author, container-title, URL, issued) plus skill-internal fields (status, version, pdf_path, arxiv_id, etc.) that pandoc ignores. This means the file is directly usable as pandoc --bibliography references.yaml with no conversion step.

A references: root key is also accepted, because that is the form pandoc's own documentation shows and what an existing pandoc bibliography usually looks like; so is the skill's earlier papers: key. Whichever shape a file arrives in is preserved when the skill writes it back, so pointing the skill at a bibliography you already maintain does not reformat it. Any other shape is refused rather than read as an empty database, and no command will write an empty database over a file that had entries.

Every write goes out in yq's normal form: the skill runs yq -i '.' over the file after serializing it. PyYAML and yq indent nested sequences differently, and on a bibliography of a few hundred entries that disagreement turns any hand-edit made with yq into a diff the length of the file. One tool decides the format, and it is the one the operator edits with. Without yq on PATH the database is still written and still valid, in PyYAML's style, and the skill says so on stderr — the next yq edit will reformat it whole.

An entry looks like:

- id: lee-2026
  type: article
  title: "Meta-Harness: End-to-End Optimization of Model Harnesses"
  author:
    - family: Lee
      given: Yoonho
  container-title: arXiv preprint arXiv:2603.28052
  URL: https://arxiv.org/abs/2603.28052
  issued:
    year: 2026
  arxiv_id: "2603.28052"
  version: 1
  status: downloaded
  pdf_path: pdfs/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.pdf
  md_path: papers/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md

The id field is a pandoc citation key (used as @lee-2026 in markdown), generated from the first author's family name and the year — just <family>-<year>, with a letter suffix on collision (lee-2026a). It is deliberately short; the richer provenance lives in the file names. The arxiv_id field is the base arXiv identifier used for deduplication and version tracking.

Entries fetched through OpenAlex additionally carry two marker blocks that later steps use (pandoc ignores them):

  openalex_id: W2158864412
  ranking:
    cited_by: 5671            # citations TO the paper
    references_count: 52      # citations IN the paper
    fwci: 102.6               # field-weighted citation impact (1.0 = field average)
    citation_percentile: 100.0
    venue: IEEE Transactions on Software Engineering
    institutions: [Massachusetts Institute of Technology]
    institution_types: [education]
    retrieved: 2026-07-15     # counts age; they are dated
  discovery:
    method: survey-references # seed-search | survey-references | forward-citations | author-drill
    via: "Wang et al. 2024 autonomous-agents survey"

ranking answers "how important, by whose measure, as of when"; discovery answers "how did this paper enter the corpus". Both persist so the reading step can prioritize and the summaries can cite them.

Running the scripts

The Python scripts run in a pixi-managed environment that ships beside the skill: pixi.toml and pixi.lock sit at the root of the agent directory (the parent of this skill's skills/ folder). Provision it once per machine — the agent does this on repo open — with the preflight:

<agent-dir>/scripts/ensure-env.sh

Then run every script through pixi. For brevity the commands below write $RUN for the wrapper:

RUN="pixi run --manifest-path <skill>/../../pixi.toml python"

where <skill>/../../ resolves to the agent directory holding pixi.toml. This replaces the older pip install --user path — no packages are installed into the ambient interpreter.

The workflow

0. Reconcile the database

Run this first, every time the skill is invoked. The goal is a complete database in the correct naming convention before anything else happens:

$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile

reconcile (the same command as repair) walks the whole database and the pdfs/ directory and makes disk and db agree. The corpus is additive (GH-234): reconcile never renames, moves, or deletes an existing file by default. Its default duties:

• an entry missing its markdown is converted;
• pdf/markdown/summary files on the older naming scheme are reported as rename candidates; the migration itself runs only under --rename (paths updated in the db; citation ids left intact), and even then a file living outside the skill's own subdirectory for its field — a curated fulltext/ tree, say — is never touched: the skill did not put it there and does not own its layout. Every executed rename prints its old -> new pair;
• a PDF in pdfs/ that no entry references is imported — its metadata is recovered from an arXiv id in the filename (fetched from arXiv), or, failing that, from the PDF's embedded title/author (recorded status: needs-review with year nd). Anything unrecoverable is listed in <db-dir>/unregistered-pdfs.md with a ready-to-run ingest command.

It is idempotent and prints counts (converted, renamed, rename_candidates, curated_skipped, collisions, imported, needs_review, unregistered). In the additive default an orphan whose derived name would land on a DIFFERENT live entry's parked markdown is refused with the reason (the GH-31 hazard, revived by opt-in renames and re-guarded); the same paper reattaching its lost PDF still rewrites its own markdown. A non-zero collisions means two entries resolved to the same filename — usually a duplicated citation id — so the rename was refused and those files were left on their old names rather than one overwriting the other. Fix the duplicate and re-run.

The same count covers a refused import: an orphan that identifies as a paper the database already holds is a duplicate copy, not a new paper, so it is left in pdfs/ under its own name instead of being renamed over the copy already there. Delete it, or correct the entry it duplicates. To register an unidentified PDF by hand, give ingest the metadata and it creates the entry:

$RUN <skill>/scripts/scholar.py --db <db-path> ingest --file pdfs/<file>.pdf \
  --title "Exact Title" --authors "Given Family" --year 2024

After reconciling, refresh the Obsidian tags and source links on every summary (safe to run each time — it only changes what is stale):

$RUN <skill>/scripts/keywords.py --db <db-path> tag

1. Understand the current work first

Before searching, read what's in the working directory — a draft paper, notes, an outline, existing summaries. The search is only as good as its query, and the query should come from the actual problem the user is working on, not just the standing topic list. If the directory is empty or the intent is unclear, ask the user what angle they care about (one question, then proceed).

2-3. Search, dedupe, fetch

Per-backend mechanics — source order, dedupe rules, and how each fetch path behaves — are in search-and-fetch.md. In short: search the hierarchical backends, dedupe against references.yaml before fetching anything, fetch the PDF where a direct link exists, and record a metadata-only entry where it does not.

4. Read and summarize

When several papers await reading, use the ranking markers to order them: pivotal first (high fwci/cited_by), and read a survey before the papers discovered through it (discovery.via tells you which those are).

Read the markdown conversion that fetch produced — its path is in the md_path field of the fetch output and the db entry (under <db-dir>/papers/). If the markdown file is missing (e.g. an older fetch before markdown conversion was available), run repair first to regenerate it. Then write a summary file following references/summary-template.md exactly. The summary's job is to let the user decide, in two minutes, whether to cite the paper — so the "Relevance to the current work" section carries the weight. Tie findings back to the current draft and, where it fits, to spindle's state-machine / declarative-agent view. Prefer the paper's own numbers over adjectives. An honest "low relevance" beats a stretch.

Name the summary with the same stem as the paper's PDF and markdown so its source is obvious: <db-dir>/summaries/<Family>-<Year>-<title-slug>-<source>-<id>.md (the stem is the basename of the entry's md_path, with the summaries/ directory). If you are unsure of the exact stem, run repair afterward — it renames any summary to match.

5. Record it

Close the loop so the database reflects reality:

$RUN <skill>/scripts/arxiv.py --db <db-path> record --id 2310.12345 \
  --summary-file summaries/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md \
  --topics llm agents fsm declarative-agents \
  --relevance "One line on why it matters to this work."

This flips the entry to status: summarized. The database is now the single source of truth: re-running a search later will mark these papers known and skip them.

Then apply Obsidian tags and source links to the new summaries:

$RUN <skill>/scripts/keywords.py --db <db-path> tag

This walks the db and, for every summary, writes a normalized tags: block into the frontmatter — merging the paper's declared keywords (or a frequency fallback when it declares none) with the entry's topics, plus a paper root tag — and adds a **Source:** line linking the local PDF, the converted full text, and the source URL. It also drops a one-time PDF back-link at the top of each converted paper. Tags a human added are preserved; --dry-run previews without writing; a second run is a no-op. Because it walks the whole db, it also tags any older summary that predates this step, so a corpus converges.

6. Report back

Summarize what was found: how many candidates, how many were new vs. already known, which were summarized, and a one-line takeaway per paper with a link to its summary file. Point out the two or three most relevant to the current work. If any papers came back metadata-only, run scholar.py pending and hand the user downloads-needed.md so they can fetch those PDFs and return them via ingest.

Repair / reconcile

repair and reconcile are the same command — the database reconcile run at step 0. Run it any time to bring disk and db back into agreement:

$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile

It walks every entry, re-converts any missing md_path from the PDF on disk, migrates legacy text_path entries, and renames any pdf/markdown/summary still on the older naming scheme to the human-friendly stem (updating pdf_path, md_path, summary_file; citation id values left unchanged). It then scans pdfs/ for files no entry references and imports them (arXiv-id recovery, then embedded-metadata needs-review, then the unregistered-pdfs.md list). It is idempotent — correctly-named, tracked files are left alone — and no step ever moves a file onto one that already exists: a rename or an import that would replace another paper is refused, counted as a collision, and explained on stderr. Prints a JSON summary (checked, converted, renamed, collisions, imported, needs_review, unregistered, skipped).

Dependencies

The pixi environment (see "Running the scripts") supplies PyYAML (required), pymupdf4llm (PDF-to-markdown conversion preserving headings, tables, and math), and pypdf (the plain-text fallback). It is provisioned by ensure-env.sh; no pip install is needed. If the conversion libraries were somehow absent, fetch still downloads the PDF and just skips the conversion. yq (the Go implementation, brew install yq) normalizes the database on write; it is optional, and its absence costs formatting, not correctness. Everything else is Python stdlib plus the arXiv public API — no key needed. Be a good citizen: the script already retries with backoff; don't hammer the API with huge --max values in a tight loop.

OpenAlex needs no key — set OPENALEX_MAILTO (any email) for the polite pool's higher rate limits. Google Scholar search requires a SerpAPI key (same key as the idea-factory job-search skill). Semantic Scholar needs no key — the public Graph API is open, though shared and rate-limited; an optional SEMANTIC_SCHOLAR_API_KEY (or --api-key) raises the limit.