Daily Arxiv

/daily-arxiv

Pulls new papers from arXiv RSS daily, automatically assesses relevance based on research directions and concepts in the wiki, calls /ingest to fully incorporate highly relevant papers into the wiki, detects SOTA updates, and generates a digest log. Supports cron-scheduled automatic execution as well as manual triggering.

Inputs

• --hours N: pull papers from the last N hours (default 24)
• --max-ingest N: maximum papers to ingest per run (default 5, prevents wiki overload)
• --dry-run: generate digest only, do not execute ingest
• --categories: override default arXiv categories (default: cs.LG cs.CV cs.CL cs.AI stat.ML)

Outputs

• raw/discovered/{slug}/ or raw/discovered/{slug}.pdf — fetched source artifact for each auto-ingested paper
• wiki/papers/{slug}.md — highly relevant paper pages (created via /ingest)
• Corresponding concepts/, people/, claims/ pages (created via /ingest)
• Updated wiki/topics/*.md — SOTA tracker annotations (if SOTA update detected)
• Updated wiki/graph/ — edges.jsonl, context_brief.md, open_questions.md (maintained via /ingest)
• Updated wiki/index.md and wiki/log.md

Wiki Interaction

Reads

• wiki/topics/*.md — extract Overview keywords and SOTA tracker, used for relevance scoring and SOTA detection
• wiki/concepts/*.md — extract Definition keywords, assist relevance scoring
• wiki/index.md — check whether a paper is already collected (deduplicate by arXiv URL)
• wiki/papers/*.md — check whether an arxiv ID already exists
• wiki/graph/open_questions.md — prioritize ingesting papers that fill knowledge gaps

Writes

• wiki/papers/{slug}.md — CREATE via /ingest
• wiki/concepts/{slug}.md — CREATE/EDIT via /ingest
• wiki/people/{slug}.md — CREATE/EDIT via /ingest
• wiki/claims/{slug}.md — CREATE/EDIT via /ingest
• wiki/topics/{slug}.md — EDIT (SOTA tracker annotations)
• wiki/graph/edges.jsonl — APPEND via /ingest
• wiki/graph/context_brief.md — REBUILD (once at the end)
• wiki/graph/open_questions.md — REBUILD (once at the end)
• wiki/index.md — EDIT via /ingest
• wiki/log.md — APPEND

Graph edges created

• All edges created by /ingest (paper → concept, paper → claim, etc.)

Workflow

Pre-conditions: confirm the working directory is the wiki project root (directory containing wiki/, raw/, tools/). Set WIKI_ROOT=wiki/.

Step 1: Pull arXiv RSS + Trending Papers

1. Run fetch_arxiv.py to get the new paper list:

   python3 tools/fetch_arxiv.py --hours <hours> -o /tmp/arxiv_feed.json
   

2. Fetch DeepXiv trending papers (past 7 days):

   python3 tools/fetch_deepxiv.py trending --days 7 --limit 20
   

   Merge trending papers into the candidate list (deduplicated by arxiv_id); trending papers receive extra attention in subsequent scoring. If DeepXiv is unavailable: skip this sub-step, use RSS results only.
3. Parse results to obtain the paper list (title, abstract, authors, arxiv_url, arxiv_id, category)
4. Deduplication: read wiki/index.md, skip papers whose arXiv URL is already in the wiki. Also check existing arxiv IDs in the wiki/papers/ directory.
5. If no new papers, skip directly to Step 6 to generate an empty digest.

Step 2: Build Relevance Context + DeepXiv Enhancement

1. Read wiki/topics/*.md and extract for each topic:
   • Core keywords from the Overview paragraph
   • Open problems / Research gaps list
   • Current best results from the SOTA tracker
2. Read wiki/concepts/*.md and extract for each concept:
   • Key terms from the Definition paragraph
   • tags list
3. Read wiki/graph/open_questions.md for the current knowledge gap list
4. Synthesize a "research direction summary" (≤ 2000 characters) containing: core topics, active concepts, gaps to fill
5. DeepXiv TLDR enhancement (optional): for each new paper, fetch an AI summary and keywords to improve scoring quality:

   python3 tools/fetch_deepxiv.py brief <arxiv_id>
   

   Supplement the original abstract with the returned tldr and keywords to help the LLM judge relevance more precisely. If DeepXiv is unavailable: use only the RSS original title + abstract for scoring (fallback to original behavior).

Step 3: Relevance Scoring

For each new paper, LLM assesses relevance based on title and abstract vs. the research direction summary:

Bonus rules (can promote a score of 2 to 3):

• Paper directly addresses a knowledge gap in open_questions.md → +1
• Paper's benchmark may update the SOTA tracker → +1 (capped at 3)

Batch scoring: submit all papers' title+abstract to the LLM in a single call and return scores as JSON. Avoid per-paper calls.

Step 4: Auto-Ingest High-Priority Papers (with checkpoint resume)

1. Filter papers with relevance = 3, sorted by the following priority:
   • Papers that fill gap_map gaps first
   • Papers with higher citation counts first (if abstract mentions SOTA results)
2. Load checkpoint (skip already-completed papers if one exists):

   python3 tools/research_wiki.py checkpoint-load wiki/ "daily-arxiv-{date}"
   

3. Take the first --max-ingest papers (default 5). For each selected paper:
   • Download the source artifact into raw/discovered/ first:

     python3 tools/init_discovery.py download --raw-root raw --arxiv-id <arxiv_id> --title "<title>"
     

   • Pass the returned canonical_ingest_path from raw/discovered/ into /ingest, not the bare arXiv URL
   • /ingest completes the full wiki incorporation flow (paper + concepts + people + claims + cross-refs + graph)
   • After each success, record checkpoint:

     python3 tools/research_wiki.py checkpoint-save wiki/ "daily-arxiv-{date}" "{arxiv_id}"
     

   • On failure, mark and continue:

     python3 tools/research_wiki.py checkpoint-save wiki/ "daily-arxiv-{date}" "{arxiv_id}" --failed
     

4. If --dry-run, skip both the raw/discovered/ download and the actual ingest; mark "would ingest" in the digest
5. After all done, clear checkpoint:

   python3 tools/research_wiki.py checkpoint-clear wiki/ "daily-arxiv-{date}"
   

Step 5: SOTA Detection and Update

1. For each paper ingested in Step 4, check the benchmark numbers in its Results section
2. Compare benchmarks against the ## SOTA tracker in the corresponding wiki/topics/ page
3. If the paper's results beat the current SOTA record:
   • Append/update an entry in the topic page's ## SOTA tracker:

     - **{benchmark_name}**: {score} ← [[{paper-slug}]] ({year}) [previously: {old_score}]
     

   • Set sota_updated for that topic to today's date
4. If SOTA updates are detected, highlight them in the digest

Step 6: Generate Digest and Write to Log

1. Rebuild graph derived files (only if any ingest happened):

   python3 tools/research_wiki.py rebuild-context-brief wiki/
   python3 tools/research_wiki.py rebuild-open-questions wiki/
   

2. Append digest to wiki/log.md:

   python3 tools/research_wiki.py log wiki/ "daily-arxiv | {N_ingested} ingested, {N_relevant} relevant / {N_total} total"
   

3. Append detailed digest below the current day's log entry:

   ### High Priority (ingested)
   - [[paper-slug]] — {title} ({one-line insight})
   
   ### Worth Watching (relevance = 2)
   - {title} — {arxiv_url} — {one-line summary}
   
   ### Trending This Week (from DeepXiv)
   - {title} — {arxiv_id} — {tweets} tweets, {views} views
   
   ### SOTA Updates
   - {topic}: {benchmark} new record by [[paper-slug]]
   
   <details>
   <summary>Weakly Relevant ({K} papers)</summary>
   
   - {title} — {arxiv_url}
   
   </details>
   

Step 7: Report to User

Output summary:

• Total papers scanned / count after deduplication
• Distribution across relevance levels
• List of ingested papers (with slug links)
• List of SOTA updates (if any)
• Recommended manual ingest candidates (top 3 most notable from relevance = 2)
• Next run time reminder

Constraints

• Only ingest papers with relevance >= 3: leave the rest for user judgment, do not auto-create wiki pages
• At most --max-ingest papers per run (default 5): prevents single-run wiki overload
• /daily-arxiv is raw-read-only except raw/discovered/ for auto-ingested papers: never write to raw/papers/, raw/tmp/, raw/notes/, or raw/web/
• graph/ maintained via tools only: do not manually edit graph files
• Bidirectional links: guaranteed by /ingest
• Deduplication must be strict: double-check by both arxiv_url and arxiv_id
• Batch scoring: one LLM call to score all papers, no per-paper calls
• Digest stays concise: see individual papers pages for details; at most one line per paper in the digest
• log.md is append-only: use python3 tools/research_wiki.py log to append

Error Handling

• DeepXiv API unavailable: fall back to pure RSS mode (original behavior). Trending section omitted from digest; scoring uses only raw RSS data. Note DeepXiv unavailability in the report.
• RSS fetch fails: report network error, suggest user check network and retry. Do not modify the wiki.
• Partial ingest failures: keep completed ingests, mark failed papers in the report, suggest user manually /ingest <url>.
• wiki directory does not exist: prompt user to run /init first.
• Empty RSS results: normal situation (fewer papers on holidays/weekends), generate empty digest without error.
• SOTA comparison fails: if benchmark format does not match, skip and note in report.

Dependencies

Skills（via Skill tool）

• /ingest — full paper incorporation flow (called in Step 4)

Tools（via Bash）

• python3 tools/fetch_arxiv.py --hours <N> -o <path> — pull arXiv RSS
• python3 tools/fetch_deepxiv.py trending --days 7 --limit 20 — fetch trending papers
• python3 tools/fetch_deepxiv.py brief <arxiv_id> — fetch paper TLDR and keywords
• python3 tools/init_discovery.py download --raw-root raw --arxiv-id <id> --title "<title>" — download selected papers into raw/discovered/
• python3 tools/research_wiki.py rebuild-context-brief wiki/ — rebuild compressed context
• python3 tools/research_wiki.py rebuild-open-questions wiki/ — rebuild knowledge gap map
• python3 tools/research_wiki.py log wiki/ "<message>" — append log

External APIs

• arXiv RSS (via tools/fetch_arxiv.py)
• DeepXiv API (via tools/fetch_deepxiv.py, optional; graceful fallback when unavailable)

Scheduling

• Can be scheduled for daily automatic execution via CronCreate:

  CronCreate: schedule "/daily-arxiv" daily at 08:00