Academic Paper Search

Academic Paper Search

A unified toolkit for discovering, retrieving metadata, and downloading academic papers across all major sources. The right API depends on the task — this skill routes to the best source.

Decision Tree: Which API to Use

What is needed?
│
├─ SEARCH for papers on a topic
│  ├─ arXiv preprints (CS/AI/Physics/Econ) ──→ arXiv API
│  ├─ NBER working papers (economics) ───────→ NBER API
│  ├─ SSRN working papers (social science) ──→ OpenAlex (primary) + CrossRef prefix:10.2139
│  ├─ Published journal articles ────────────→ CrossRef API or OpenAlex
│  └─ Broad multi-source search ─────────────→ OpenAlex API
│
├─ GET METADATA for a known paper
│  ├─ Have DOI ──→ CrossRef (structured) or OpenAlex (richer)
│  ├─ Have arXiv ID ──→ arXiv API
│  ├─ Have NBER number ──→ NBER API
│  └─ Have SSRN abstract ID ──→ OpenAlex DOI lookup (DOI = 10.2139/ssrn.{id})
│
├─ FIND/DOWNLOAD PDF
│  ├─ Have DOI ──→ Unpaywall API (check OA) → OpenAlex (alt PDF links)
│  ├─ arXiv paper ──→ Direct: https://arxiv.org/pdf/{id}
│  ├─ NBER paper ──→ Direct: https://www.nber.org/system/files/working_papers/w{n}/w{n}.pdf
│  └─ SSRN paper ──→ Unpaywall (~30%) or institutional repos (EconStor, IZA)
│
└─ IMPORT TO ZOTERO
   └─ Always use zotero_add_by_doi (best metadata + auto PDF via Unpaywall cascade)

1. OpenAlex API (Best All-Round)

Free, no API key needed. Covers 250M+ works including SSRN, NBER, arXiv, all journals. Add mailto=your@email.com for polite pool (faster responses).

Search by Source

# SSRN papers (source ID: S4210172589, NOT S4306400806 which is Europe PMC)
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210172589,default.search:social+preferences,publication_year:2024-2026&per_page=25&mailto=you@university.edu"

# NBER papers
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210174556,default.search:behavioral+economics&per_page=25&mailto=you@university.edu"

Search by Topic (All Sources)

curl -s "https://api.openalex.org/works?search=network+formation+game+theory&filter=publication_year:2024-2026&per_page=25&sort=cited_by_count:desc&mailto=you@university.edu"

DOI Lookup

curl -s "https://api.openalex.org/works/doi:10.2139/ssrn.4812743?mailto=you@university.edu"

Python Pattern

import requests

def openalex_search(query, source_id=None, year_from=2024, year_to=2026, per_page=25):
    """Search OpenAlex. source_id: S4210172589 (SSRN), S4210174556 (NBER), etc."""
    filters = [f"default.search:{query}", f"publication_year:{year_from}-{year_to}"]
    if source_id:
        filters.insert(0, f"primary_location.source.id:{source_id}")
    params = {
        "filter": ",".join(filters),
        "per_page": per_page,
        "mailto": "you@university.edu"
    }
    data = requests.get("https://api.openalex.org/works", params=params).json()
    results = []
    for w in data.get("results", []):
        pdf_urls = [loc["pdf_url"] for loc in w.get("locations", []) if loc.get("pdf_url")]
        results.append({
            "title": w["title"],
            "doi": w.get("doi", "").replace("https://doi.org/", ""),
            "year": w.get("publication_year"),
            "authors": [a["author"]["display_name"] for a in w.get("authorships", [])],
            "cited_by": w.get("cited_by_count", 0),
            "pdf_urls": pdf_urls,
            "landing_url": (w.get("primary_location") or {}).get("landing_page_url", ""),
        })
    return {"total": data.get("meta", {}).get("count", 0), "results": results}

Key Source IDs

Full reference: See references/journal_identifiers.md for a comprehensive table of 77 verified OpenAlex Source IDs and CrossRef ISSNs covering Top 5 economics, AEA journals, finance, management, marketing, accounting, IS, and 15+ field journal categories.

Warning: OpenAlex source IDs are opaque and not guessable. Always verify via https://api.openalex.org/sources?search=journal+name before using a new ID. For verified IDs of 77 journals, consult references/journal_identifiers.md.

PDF from OpenAlex

primary_location.pdf_url is null for SSRN papers (Elsevier blocks it). Check locations[].pdf_url for third-party repos — ~24% of SSRN papers have PDFs via EconStor, IZA, MPRA, etc. These are real, downloadable PDFs.

2. CrossRef API (Best for DOI Metadata)

Free, no key needed. Add mailto= for 50 req/sec (vs 1 req/sec without).

Search with SSRN DOI Prefix

# All SSRN papers matching a query
curl -s "https://api.crossref.org/works?query=%22social+preferences%22&filter=prefix:10.2139,from-pub-date:2024-01-01&rows=25&sort=relevance&mailto=you@university.edu"

Search within a Journal

# See references/journal_identifiers.md for 77 verified ISSNs
# AER: 0002-8282, QJE: 0033-5533, JPE: 0022-3808, Econometrica: 0012-9682, REStud: 0034-6527
curl -s "https://api.crossref.org/journals/0002-8282/works?query=auction+mechanism&rows=10&mailto=you@university.edu"

DOI Lookup

curl -s "https://api.crossref.org/works/10.1257/aer.20171330?mailto=you@university.edu"

DOI Construction Rules

These predictable DOI formats allow constructing DOIs from paper IDs without making extra API calls.

Limitations

CrossRef returns title, authors, DOI, date, journal — but no abstracts for SSRN papers (Elsevier doesn't submit them to CrossRef). Use OpenAlex for richer metadata.

3. arXiv API (Preprints — CS/AI/Physics/Math/Econ)

Free, no key needed. Returns Atom XML. Rate limit: ~1 request per 3 seconds.

curl -s "https://export.arxiv.org/api/query?search_query=abs:social+preferences+AND+(cat:econ.TH+OR+cat:econ.GN)&max_results=20&sortBy=submittedDate&sortOrder=descending"

Search Prefixes

all: (full text), ti: (title), au: (author), abs: (abstract), cat: (category). Boolean: AND, OR, ANDNOT. Exact phrase: %22social+preferences%22.

Economics Categories

econ.TH (Theory), econ.GN (General), econ.EM (Econometrics).

Caveat: The cat: filter in search_query does not always reliably restrict results to specified categories — arXiv may still return papers from other fields. Verify category matches in results and filter client-side if needed.

Direct PDF

https://arxiv.org/pdf/{id}          # e.g. https://arxiv.org/pdf/2401.12345
https://arxiv.org/abs/{id}          # abstract page

4. NBER API (Economics Working Papers)

Free, no key needed. Returns JSON.

# Search
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&q=social+preferences"

# Filter by Program (replace _/_ with programs/{name})
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/programs/Labor+Studies/search?page=1&perPage=20&q=minimum+wage"

# New this week
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&newThisWeek=true"

Programs

Labor Studies, Economic Fluctuations and Growth, Industrial Organization, Public Economics, Development Economics, International Finance and Macroeconomics, Corporate Finance, Asset Pricing, Health Economics, etc.

Response Processing

Authors field contains HTML <a> tags — strip with: re.sub(r'<[^>]+>', '', text)

No built-in date filter — API sorts by relevance. Filter client-side by displaydate.

Direct PDF

https://www.nber.org/system/files/working_papers/w{number}/w{number}.pdf

5. Unpaywall API (Open Access PDF Finder)

Free, no key needed. Email as query param. 100,000 requests/day.

curl -s "https://api.unpaywall.org/v2/10.2139/ssrn.4812743?email=you@university.edu" \
  | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'OA: {d[\"is_oa\"]} ({d[\"oa_status\"]})')
pdf = (d.get('best_oa_location') or {}).get('url_for_pdf')
print(f'PDF: {pdf or \"none\"}')
# Check all locations for alternative PDFs
for loc in d.get('oa_locations', []):
    if loc.get('url_for_pdf'):
        print(f'Alt PDF: {loc[\"url_for_pdf\"]} ({loc[\"host_type\"]})')
"

OA Status Values

Platform-Specific PDF Availability

For SSRN papers where Unpaywall returns url_for_pdf=null, check oa_locations[] for institutional repositories (EconStor, university repos) — follow their landing pages to find PDF links.

6. Semantic Scholar API (Citations & Impact)

Free, no key needed. Rate limit: 100 requests per 5 minutes. Warning: As of 2026, Semantic Scholar may return 429 errors frequently. Consider registering for an API key at https://www.semanticscholar.org/product/api for higher limits. Add header: x-api-key: YOUR_KEY.

# Search
curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=social+preferences+experiment&fields=title,year,authors,citationCount,externalIds,venue&limit=20&year=2024-2026"

# DOI lookup
curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1257/aer.20171330?fields=title,year,authors,citationCount,abstract"

Field of Study Filter

fieldsOfStudy=Economics,Sociology,Political+Science,Business,Psychology

Warning: The Economics filter is very strict and may return 0 results. Often better to search without it and filter manually.

When to Use

Best for: citation counts, finding related papers, author impact metrics. Not best for: SSRN-specific search (poor coverage), bulk downloading.

7. Tavily Search (Fallback for Any Source)

When APIs do not cover the need, Tavily can search any academic site. Particularly useful for SSRN (no public API) and as a fallback for any source.

# Via MCP tool
mcp__tavily__tavily-search({
    "query": 'site:ssrn.com "social preferences" economics 2024 2025',
    "search_depth": "advanced",
    "max_results": 20,
    "include_domains": ["ssrn.com", "papers.ssrn.com"]
})

# Also works for NBER, RePEc, etc.
mcp__tavily__tavily-search({
    "query": 'site:nber.org "network formation" working paper',
    "include_domains": ["nber.org"]
})

Recommended Workflows

Workflow A: Comprehensive Topic Search

For finding all recent papers on a topic across multiple sources:

1. OpenAlex: broad search (all sources, richest metadata)
2. CrossRef prefix:10.2139: catch SSRN papers OpenAlex might miss
3. NBER API: NBER-specific search with program filtering
4. arXiv API: preprints in CS/AI/Physics/Econ
5. Tavily: sweep for anything the APIs missed
6. Deduplicate by DOI

Workflow B: SSRN Paper Search

SSRN has no public API. Use this cascade:

1. OpenAlex (filter: source.id:S4210172589) — best: 1 API call, rich metadata
2. CrossRef (filter: prefix:10.2139) — backup: structured DOI search
3. Tavily (include_domains: ssrn.com) — catches papers not yet in OpenAlex/CrossRef

Workflow C: PDF Download Pipeline

Try sources in this order (most reliable first):

1. arXiv direct: https://arxiv.org/pdf/{id}
2. NBER direct: https://www.nber.org/system/files/working_papers/w{n}/w{n}.pdf
3. Unpaywall: check best_oa_location.url_for_pdf via DOI
4. OpenAlex: check locations[].pdf_url for third-party repo PDFs
5. Institutional repos: follow Unpaywall oa_locations landing pages
6. SSRN direct: construct Delivery.cfm URL (may be Cloudflare-blocked)

Workflow D: Zotero Import

Always prefer zotero_add_by_doi over zotero_add_by_url:

• Better metadata (correct entry types, complete fields)
• Higher PDF download success (Zotero uses Unpaywall → arXiv → S2 → PMC cascade)
• DOI can be constructed for SSRN (10.2139/ssrn.{id}) and NBER (10.3386/w{number})

Rate Limits & Best Practices