Arxiv

arXiv Research

Search and retrieve academic papers from arXiv via their free REST API, then enrich with citation data from Semantic Scholar. No API key required for either service at baseline rates.

When to Use

• Trigger keywords: "arxiv", "paper", "academic search", "citation count", "BibTeX", "related papers", "research", "preprint", "Semantic Scholar"
• You need to find papers on a topic, by an author, or in a specific category
• You need citation counts, reference lists, or paper recommendations
• You need to generate a BibTeX entry from an arXiv ID
• You need to read a paper's abstract or full PDF content

Prerequisites

• curl available on PATH (Windows: curl.exe ships with Windows 10+; in PowerShell use curl.exe to avoid the Invoke-WebRequest alias)
• python3 (or python) for XML parsing and JSON formatting — no third-party dependencies, stdlib only
• web_extract tool available for reading abstract pages and PDFs
• No API keys required for arXiv or Semantic Scholar baseline rates

Windows / PowerShell notes:

• Use curl.exe instead of curl to bypass the PowerShell alias for Invoke-WebRequest
• For multi-line Python snippets in PowerShell, use a here-string (@' … '@) or write to a temp .py file and run python file.py
• Alternatively, use the helper script scripts/search_arxiv.py which avoids inline quoting issues entirely

Procedure

1. Search arXiv by keyword

# Basic keyword search (Linux/macOS)
curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"

# PowerShell equivalent
curl.exe -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"

2. Get clean, parsed output

Use the helper script for the simplest path:

python scripts/search_arxiv.py "GRPO reinforcement learning"
python scripts/search_arxiv.py "transformer attention" --max 10 --sort date
python scripts/search_arxiv.py --author "Yann LeCun" --max 5
python scripts/search_arxiv.py --category cs.AI --sort date
python scripts/search_arxiv.py --id 2402.03300
python scripts/search_arxiv.py --id 2402.03300,2401.12345

No dependencies — uses only Python stdlib.

Alternatively, parse XML inline:

curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5&sortBy=submittedDate&sortOrder=descending" | python3 -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom'}
root = ET.parse(sys.stdin).getroot()
for i, entry in enumerate(root.findall('a:entry', ns)):
    title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
    arxiv_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
    published = entry.find('a:published', ns).text[:10]
    authors = ', '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
    summary = entry.find('a:summary', ns).text.strip()[:200]
    cats = ', '.join(c.get('term') for c in entry.findall('a:category', ns))
    print(f'{i+1}. [{arxiv_id}] {title}')
    print(f'   Authors: {authors}')
    print(f'   Published: {published} | Categories: {cats}')
    print(f'   Abstract: {summary}...')
    print(f'   PDF: https://arxiv.org/pdf/{arxiv_id}')
    print()
"

3. Use search query syntax

Boolean operators:

# AND (default when using +)
search_query=all:transformer+attention

# OR
search_query=all:GPT+OR+all:BERT

# AND NOT
search_query=all:language+model+ANDNOT+all:vision

# Exact phrase
search_query=ti:"chain+of+thought"

# Combined
search_query=au:hinton+AND+cat:cs.LG

4. Sort and paginate

# Latest 10 papers in cs.AI
curl -s "https://export.arxiv.org/api/query?search_query=cat:cs.AI&sortBy=submittedDate&sortOrder=descending&max_results=10"

5. Fetch specific papers by ID

# Single paper
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300"

# Multiple papers
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300,2401.12345,2403.00001"

6. Generate BibTeX

curl -s "https://export.arxiv.org/api/query?id_list=1706.03762" | python3 -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
root = ET.parse(sys.stdin).getroot()
entry = root.find('a:entry', ns)
if entry is None: sys.exit('Paper not found')
title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
authors = ' and '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
year = entry.find('a:published', ns).text[:4]
raw_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
cat = entry.find('arxiv:primary_category', ns)
primary = cat.get('term') if cat is not None else 'cs.LG'
last_name = entry.find('a:author', ns).find('a:name', ns).text.split()[-1]
print(f'@article{{{last_name}{year}_{raw_id.replace(\".\", \"\")},')
print(f'  title     = {{{title}}},')
print(f'  author    = {{{authors}}},')
print(f'  year      = {{{year}}},')
print(f'  eprint    = {{{raw_id}}},')
print(f'  archivePrefix = {{arXiv}},')
print(f'  primaryClass  = {{{primary}}},')
print(f'  url       = {{https://arxiv.org/abs/{raw_id}}}')
print('}')
"

7. Read paper content

# Abstract page (fast, metadata + abstract)
web_extract(urls=["https://arxiv.org/abs/2402.03300"])

# Full paper (PDF → markdown via Firecrawl)
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])

For local PDF processing, see the ocr-and-documents skill.

8. Enrich with Semantic Scholar (citations, references, recommendations)

Semantic Scholar returns JSON — pipe through python3 -m json.tool for readability.

Get paper details + citation counts:

# By arXiv ID
curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300?fields=title,authors,citationCount,referenceCount,influentialCitationCount,year,abstract" | python3 -m json.tool

# By DOI
curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1234/example?fields=title,citationCount"

Get citations OF a paper (who cited it):

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/citations?fields=title,authors,year,citationCount&limit=10" | python3 -m json.tool

Get references FROM a paper (what it cites):

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/references?fields=title,authors,year,citationCount&limit=10" | python3 -m json.tool

Search papers (returns JSON):

curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=GRPO+reinforcement+learning&limit=5&fields=title,authors,year,citationCount,externalIds" | python3 -m json.tool

Get paper recommendations:

curl -s -X POST "https://api.semanticscholar.org/recommendations/v1/papers/" \
  -H "Content-Type: application/json" \
  -d '{"positivePaperIds": ["arXiv:2402.03300"], "negativePaperIds": []}' | python3 -m json.tool

Author profile:

curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=Yann+LeCun&fields=name,hIndex,citationCount,paperCount" | python3 -m json.tool

Useful Semantic Scholar fields:

title, authors, year, abstract, citationCount, referenceCount, influentialCitationCount, isOpenAccess, openAccessPdf, fieldsOfStudy, publicationVenue, externalIds (contains arXiv ID, DOI, etc.)

9. Complete research workflow

1. Discover: python scripts/search_arxiv.py "your topic" --sort date --max 10
2. Assess impact: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID?fields=citationCount,influentialCitationCount"
3. Read abstract: web_extract(urls=["https://arxiv.org/abs/ID"])
4. Read full paper: web_extract(urls=["https://arxiv.org/pdf/ID"])
5. Find related work: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID/references?fields=title,citationCount&limit=20"
6. Get recommendations: POST to Semantic Scholar recommendations endpoint
7. Track authors: curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=NAME"

Common Categories

Full list: https://arxiv.org/category_taxonomy

Rate Limits

Pitfalls

• arXiv returns Atom XML, not JSON. You must parse XML — use the helper script scripts/search_arxiv.py or the inline Python snippet. Do not attempt json.tool on arXiv responses.
• PowerShell curl alias. In PowerShell, curl is aliased to Invoke-WebRequest by default. Use curl.exe explicitly, or remove the alias with Remove-Item Alias:curl in your session.
• Inline Python in PowerShell. Multi-line python3 -c "..." strings break in PowerShell due to quoting differences. Prefer writing to a temp .py file or using scripts/search_arxiv.py.
• Rate limiting. arXiv allows ~1 request per 3 seconds; Semantic Scholar allows 1 request per second without an API key. Exceeding these rates results in HTTP 429 or 503 errors. Add sleep 3 between arXiv calls in loops.
• arXiv ID formats. Old-format IDs like hep-th/0601001 use a slash; new-format IDs like 2402.03300 do not. Always URL-encode the slash as %2F if passing old IDs in query parameters.
• ID versioning. arxiv.org/abs/1706.03762 always resolves to the latest version. arxiv.org/abs/1706.03762v1 points to a specific immutable version. When generating citations, preserve the version suffix you actually read to prevent citation drift — a later version may substantially change content. The API <id> field returns the versioned URL (e.g., http://arxiv.org/abs/1706.03762v7).
• Withdrawn papers. Papers can be withdrawn after submission. The <summary> field will contain a withdrawal notice (look for "withdrawn" or "retracted"). Metadata fields may be incomplete. Always check the summary before treating a result as a valid paper.
• Semantic Scholar coverage gaps. Not all arXiv papers are indexed in Semantic Scholar. If you get a 404, the paper may not yet be indexed — fall back to arXiv metadata only.
• max_results cap. The arXiv API supports up to 30000 results per query, but large result sets are slow and may time out. Use start for pagination instead of requesting everything at once.

Verification

1. Verify arXiv API is reachable:

curl -s "https://export.arxiv.org/api/query?search_query=all:test&max_results=1" | head -5

Expected: XML output beginning with <?xml and containing <feed and at least one <entry>.

2. Verify helper script works:

python scripts/search_arxiv.py "attention is all you need" --max 1

Expected: One result line with arXiv ID 1706.03762 (the "Attention Is All You Need" paper).

3. Verify Semantic Scholar is reachable:

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:1706.03762?fields=title,citationCount" | python3 -m json.tool

Expected: JSON with "title": "Attention Is All You Need" and a numeric citationCount.

4. Verify web_extract can read an abstract page:

web_extract(urls=["https://arxiv.org/abs/1706.03762"])

Expected: Markdown content containing the paper title, authors, and abstract text.

Related skills

• ocr-and-documents — for local PDF processing, OCR, and document extraction when web_extract on arXiv PDFs is insufficient or you need offline analysis.