Supplementary Search

Supplementary Search (Multi-Source)

Search additional academic sources for working papers, citation chains, and forthcoming articles that the primary search engines may miss.

Input: a research query string, optionally seed DOIs for citation chaining Output: <stem>.ris + <stem>.json (same schema as Undermind/Scholar Labs)

This is Stage 4 of a lit review pipeline. Output can be merged with Stages 1-2 (Undermind, Scholar Labs) for dedup and screening.

Quick Start

# Search all sources
python scripts/supplementary_search.py "corporate governance regulation" -o results.json --all

# Just SSRN and NBER
python scripts/supplementary_search.py "voting rules" -o results.json --ssrn --nber

# Citation chaining from prior results
python scripts/supplementary_search.py "voting rules" -o results.json \
  --citation-chain --seeds-from undermind_results.json --top-seeds 20

# Forthcoming papers with caching
python scripts/supplementary_search.py "shareholder activism" -o results.json --forthcoming --cache

Prerequisites

pip install requests beautifulsoup4 anthropic

API Keys

• SEARCHAPI_API_KEY — required for SSRN, HeinOnline, and forthcoming journal searches (uses SearchAPI.io's Google Scholar engine). Get one at searchapi.io.
• ANTHROPIC_API_KEY — required for query condensation (uses Claude Sonnet). Already needed for other pipeline stages.

Source Flags

CLI Flags

Output Schema

Same as Undermind/Scholar Labs, with source tag:

{
  "title": "Paper Title",
  "authors": "Author1, Author2",
  "year": "2024",
  "doi": "https://doi.org/10.1234/...",
  "abstract": "...",
  "journal": "Journal Name",
  "relevance": "",
  "score": "",
  "url": "https://...",
  "source": "ssrn"
}

Query Condensation

Long research prompts (~2000 chars) produce poor Google Scholar results. Before any SearchAPI call, the script uses Claude Sonnet to distill the full prompt into 3-5 short (3-6 word) queries covering distinct literature streams. Each condensed query is run per source, and results are deduplicated.

• Runs automatically unless --no-condense is passed
• --no-condense wraps the raw query in a single-element list (useful for short manual queries that don't need condensation)
• Citation chaining is unaffected (uses seed DOIs, not text queries)
• NBER also uses condensed queries (one API call per condensed query)

In the orchestrator's agent-driven flow the orchestrator passes --queries-file with agent-written queries (a Sonnet subagent by default, Opus when all_opus), bypassing condense_query entirely; the in-script Claude Sonnet condensation here is the autonomous fallback for standalone raw-query runs.

Important

• SSRN, HeinOnline, and forthcoming use SearchAPI's Google Scholar engine — requires SEARCHAPI_API_KEY env var. No Playwright or browser needed.
• Semantic Scholar API is free but rate-limited (100 req / 5 min) — the script adds polite delays automatically
• Forthcoming lists: use --cache to avoid re-querying within 24 hours
• Results are deduplicated by DOI and title before output

Troubleshooting

SSRN/Forthcoming returns no results: Check that SEARCHAPI_API_KEY is set. Inspect supplementary_debug/searchapi_*.json for raw responses.

Semantic Scholar rate limit: The script auto-throttles. If you hit limits, reduce --top-seeds or --hops.