Lit Review Orchestrator

Lit-Review Orchestrator

Run the literature-review pipeline from a single command, starting from a document that describes your article.

Input: a .tex or .docx document — a full manuscript, an abstract, or any text describing the article's content. Output: a deduplicated, relevance-screened master list (JSON + RIS), plus the extracted search plan and all intermediate stage files.

Quick Start

The orchestrator.py commands below are the autonomous fallback (reasoning through codex exec when the Codex CLI is installed, otherwise the Sonnet/DeepSeek API). When an agent runs this skill interactively, use the agent-driven flow instead; see How it runs below. That flow performs non-browser reasoning at the agent layer with no Anthropic API key, using the platform routing in docs/claude-code.md or docs/codex.md.

pip install -r requirements.txt                                  # one-time
cp lit-review-pipeline.env.example ~/.lit-review-pipeline.env    # then fill in keys

# From a full manuscript
python scripts/orchestrator.py paper.docx --output-dir ~/lit-reviews/mypaper

# From just an abstract (any .tex/.docx describing the article works)
python scripts/orchestrator.py abstract.tex --output-dir ~/lit-reviews/mypaper

# Add opt-in sources; DOI-only dedup
python scripts/orchestrator.py paper.tex --ssrn --nber --no-llm --output-dir out

# Escape hatch: run from a raw query string (skips Stage 0 extraction)
python scripts/orchestrator.py --query "dual-class shares cost of equity" --output-dir out

Pipeline

Stage 0 runs first; Stages 1, 2b, and 4a run concurrently (Stage 2 Scholar Labs joins them only with --scholarlabs, and 4b when opted in); 4c follows them; then 5, then 6.

How it runs: agent-driven (default) vs autonomous fallback

LLM work follows one routing rule: the agent-driven flow uses no Anthropic API for the pipeline's reasoning steps. Stage 0 extraction, Stage 4 query condensation, Stage 5 dedup judgments, Stage 6 screening, and keyless web search run at the agent layer. The exact model and delegation policy depends on the host agent:

• Claude Code: read docs/claude-code.md.
• Codex: read docs/codex.md.

Stage 1 (Undermind) is agent-driven as well: the host session calls the Undermind MCP tools directly (launch, poll, page), undermind_harvest.py copies the results out of the session transcript byte-for-byte, and undermind_ingest.py normalizes them. The autonomous fallback (orchestrator.py) covers unattended runs with no agent present: it fills the same reasoning seams through codex exec when the Codex CLI is installed (--llm codex, the default whenever the CLI is found) and otherwise through the scripts' Sonnet/DeepSeek API path (--llm api). It has no MCP client, so Undermind is skipped there.

Because a Python subprocess cannot spawn subagents, each reasoning script exposes an emit/ingest seam: the script does the deterministic work (parsing, candidate-pair generation, validation, enrichment, merge, all file output) and hands only the LLM step out to you in the middle. Each script also keeps its in-script Sonnet/DeepSeek API path as an autonomous fallback for unattended runs, so the same files support both interactive and unattended use.

Platform routing (agent-driven flow)

The shared pipeline uses named reasoning roles. Map those roles to the host platform before dispatching subagents or doing web-search work.

For Claude Code, the strong route is Opus (model: "opus" — always the latest Opus) and the cheaper worker route is Sonnet (model: "sonnet"). For Codex, the strong route is gpt-5.6-sol with xhigh reasoning, and lower-stakes batch judgments use the cheaper gpt-5.6-luna worker at medium or high reasoning, escalating uncertain dedup pairs to gpt-5.6-terra. See docs/claude-code.md and docs/codex.md for the complete mapping.

The GUI still emits the legacy field "all_opus". Interpret "all_opus": true as the high-accuracy platform profile: Claude Code uses Opus for all delegated reasoning, while Codex uses gpt-5.6-sol with xhigh reasoning for all delegated reasoning. When it is false or absent, follow the platform's default split.

Interactive entry (GUI)

When the skill is triggered interactively, open the settings dialog first, let the user choose the input and options, then run the agent-driven stages below honouring what it returns:

python scripts/lit_review_gui.py --config-out OUT/gui_config.json   # blocks until Run/Cancel

The window has a Browse field for the document (or a raw-query box), an output folder, the search channels as checkboxes — keyed (Undermind / Deep Research / Google Scholar checked; Scholar Labs unchecked — opt-in) and keyless (Free index search / Web search, both on) — supplementary sources (SSRN checked by default, NBER, HeinOnline) plus citation chaining, Processing (Deduplicate / Verify sources / Screen / DOI-only), and an Advanced group (Quick mode, Max chars, and a Use high-accuracy routing toggle (JSON key all_opus) that is off by default. Leave it off to run the platform's default routing split; check it to request the high-accuracy route for every delegated reasoning stage. On Run it writes the settings to --config-out and echoes them to stdout between ===LITREVIEW_CONFIG_BEGIN=== and ===LITREVIEW_CONFIG_END=== (exit 0); Cancel or closing the window exits 2 — abort the run. Parse that JSON and map it onto the stages: skip a channel set false, set output_dir, run Scholar Labs / supplementary / citation and the keyless freesearch (Stage 4e) / websearch (Stage 4d) channels when true, pass --no-verify to dedup when verify is false and --no-llm when no_llm is true, skip dedup/screen when false, pass max_chars to extraction, and, when all_opus is true, use the high-accuracy platform profile instead of the default split (see Platform routing). The GUI runs nothing itself and calls no API. Shape:

{"document":"…","query":"","output_dir":"…",
 "channels":{"undermind":true,"deepresearch":true,"scholar":true,"scholarlabs":false,"freesearch":true,"websearch":true},
 "supplementary":{"ssrn":true,"nber":false,"heinonline":false},
 "citation_chain":false,"top_seeds":20,
 "dedup":true,"verify":true,"screen":true,"no_llm":false,"quick":false,"max_chars":30000,"all_opus":false}

Agent-driven run (the default; you orchestrate)

Pick an output dir OUT. Run the deterministic stages as subprocesses and do the reasoning stages with the host platform routing described above. In Codex, spawn subagents only after explicit user authorization for parallel agent work. Substitute <doc> and the extracted <research_question>.

Stage 0: extract (strong-reasoning route):

python scripts/extract_search_plan.py <doc> --emit-prompt OUT/extract_prompt.txt -o OUT/search_plan.json
# Read OUT/extract_prompt.txt, produce the plan JSON with the platform's strong-reasoning route, write OUT/plan.json.
python scripts/extract_search_plan.py <doc> --plan-file OUT/plan.json -o OUT/search_plan.json

The plan JSON must carry extract_search_plan.py's REQUIRED_KEYS; --plan-file validates them (exit 1 on a bad plan) and writes search_plan.json/.md, scholar_queries.json, undermind_brief.txt, scholarlabs_query.txt.

Stage 1 — Undermind (agent-driven through the MCP tools; launch it first, it runs 2–5 min in the background):

list_workspaces                    # reuse a workspace for this manuscript, or create_workspace(title=<doc stem>)
launch_deep_search(workspace_id, goal=<contents of OUT/undermind_brief.txt>, name=<label>)   # returns at once
inspect_deep_searches(names=[<label>], status_only=true)        # poll from a background Monitor, not a foreground wait
inspect_deep_searches(names=[<label>], papers_only=true, detail_level="full", limit=50, offset=0 / 50 / 100 …)

# Copy the paged results out of the session transcript byte-for-byte (never retype them), then ingest:
python undermind-search/scripts/undermind_harvest.py --latest --match "<label>" --out-dir OUT
python undermind-search/scripts/undermind_ingest.py --input OUT/undermind_page_*.txt -o OUT/stage1_undermind.json

No Undermind tools in the session (server not added or not authenticated) → print UNDERMIND_DEFERRED, write [] to OUT/stage1_undermind.json, and continue. Recipe and troubleshooting: undermind-search/SKILL.md.

Stages 2b / 4a — search (subprocesses; run concurrently with Stage 1, background + Monitor). Stage 2 Scholar Labs is opt-in — run it only on request (see below):

# Opt-in only (Scholar Labs): Google rate-limits its Cite export under automation, so skip it by
# default and run this line only when asked / retrying from a fresh session:
python scholarlabs-search/scripts/scholarlabs_search.py --query-file OUT/scholarlabs_query.txt \
    --research-question "<research_question>" -o OUT/stage2_scholarlabs.json --hidden --debug-dir OUT/debug_scholarlabs
python deepresearch-search/scripts/deepresearch_search.py --query-file OUT/undermind_brief.txt \
    --research-question "<research_question>" -o OUT/stage2b_deepresearch.json --debug-dir OUT/debug_deepresearch  # Gemini Deep Research (GEMINI_API_KEY), pure subprocess
python supplementary-search/scripts/supplementary_search.py --scholar \
    --queries-file OUT/scholar_queries.json -o OUT/stage4a_scholar.json --debug-dir OUT/debug_scholar

Passing --queries-file (the agent-written queries, using the platform worker route by default and the strong route when all_opus is true) bypasses the in-script condense_query fallback in supplementary-search. For a raw-query agent run (no document, hence no Stage 0 to produce scholar_queries.json), first have the agent write that file (a short JSON array of query strings) and pass it the same way, or add --no-condense. Either route keeps the agent path free of the in-script API call.

Web search (keyless agent-driven channel, and a useful add-on alongside the keyed channels). Subagent fan-out, so the raw web text stays out of your context. This is the keyless search route in Platform routing. Emit a batched task plan, dispatch one strong-reasoning subagent per batch when parallel agents are authorized, then merge:

python websearch-search/scripts/websearch_ingest.py --emit-tasks \
    --queries-file OUT/scholar_queries.json --research-question "<research_question>" \
    --batch-size 3 -o OUT/websearch_tasks.json
# Dispatch one strong-reasoning worker per tasks[k]: hand it the system_prompt + its queries;
# each uses the platform's web-search/fetch tools over its queries and writes
# OUT/websearch_results_batch_<id>.json (only title required; never invent fields; do
# not fetch scholar.google.com). Then merge the partials:
python websearch-search/scripts/websearch_ingest.py \
    --results OUT/websearch_results_batch_*.json -o OUT/stage4d_websearch.json

This writes stage4d_websearch.json (source="websearch"), deduped by title with best-effort keyless Crossref DOI fill, which the dedup --inputs glob below picks up. The hits are real web results, so keep Stage 5b verification ON. For a few queries you can skip the fan-out and ingest a single websearch_results.json. Empty input defers (WEBSEARCH_DEFERRED). Full recipe: websearch-search/SKILL.md.

Free index search (keyless; pairs with web search for the no-key fallback). A plain keyless subprocess that searches OpenAlex / Crossref / Semantic Scholar with the Stage-0 queries:

python freesearch-search/scripts/freesearch_search.py \
    --queries-file OUT/scholar_queries.json -o OUT/stage4e_freesearch.json

This writes stage4e_freesearch.json (real index records, source set per index), which the dedup --inputs glob below also picks up. No key needed; see freesearch-search/SKILL.md.

Stage 5: dedup (platform worker route by default; strong route when all_opus):

python lit-dedup/scripts/lit_dedup.py --inputs OUT/stage[0-9]*.json --emit-pairs OUT/dedup_pairs.json -o OUT/stage5_merged.json
# Read OUT/dedup_pairs.json; for each pairs[k] = {i, j, a, b} decide if a and b are the
# same paper. Fan out across parallel platform workers for large pair sets when authorized. Write
# OUT/dedup_verdicts.json = [{"i":N,"j":N,"decision":"yes|no","confidence":"high|medium|low","rationale":"..."}].
python lit-dedup/scripts/lit_dedup.py --ingest-verdicts OUT/dedup_pairs.json OUT/dedup_verdicts.json -o OUT/stage5_merged.json

Exclude stage5_* / stage6_* from the --inputs glob. If dedup_pairs.json has no pairs, write [] to the verdicts file and still run --ingest-verdicts.

Stage 6: screen (the re-ranker; strong-reasoning route):

python lit-screen/scripts/lit_screen.py --input OUT/stage5_merged.json --query "<research_question>" \
    --emit-tasks OUT/screen_tasks.json -o OUT/stage6_screened.json
# Read OUT/screen_tasks.json = {system_prompt, query, tasks:[{index, user_message}]}.
# Score each task following system_prompt. Fan out across parallel strong-reasoning workers in batches when authorized. Write
# OUT/screen_results.json = [{"index":N,"relevance_score":1-10,"rationale":"..","paper_type":"..","identification_strategy":"..","relationship":".."}].
python lit-screen/scripts/lit_screen.py --input OUT/stage5_merged.json --ingest-results OUT/screen_results.json -o OUT/stage6_screened.json

Autonomous fallback (no agent)

python scripts/orchestrator.py <doc> --output-dir OUT              # codex exec seams when `codex` is on PATH, else API
python scripts/orchestrator.py <doc> --output-dir OUT --llm api    # force the Anthropic/DeepSeek path

Runs every stage end-to-end as subprocesses. The reasoning seams (Stage 0 extraction, Stage 4a query writing in --query mode, Stage 5 dedup verdicts, Stage 6 screening) are filled by scripts/codex_seam.py, which pipes each emitted prompt through codex exec on the ChatGPT subscription (gpt-5.6-sol/xhigh for Stages 0 and 6, gpt-5.6-luna for the batched dedup seam at high and the query seam at medium; LITREVIEW_CODEX_* overrides) and ingests the JSON through the same --plan-file / --ingest-verdicts / --ingest-results flags the agent uses. A failed Codex call falls back to the script's in-script Anthropic/DeepSeek path when ANTHROPIC_API_KEY is set (a Codex reply the ingest step rejects, or a batch left incomplete after one retry, counts as a failure); with neither available, Stage 0 aborts the run and the later seams are skipped. Low-confidence dedup verdicts are re-judged by gpt-5.6-terra; if Codex fails and no DEEPSEEK_API_KEY is set, dedup falls back to DOI-only and says so on the dashboard. Every Codex call leaves its prompt, reply, log, and (except Stage 0) schema in OUT/_codex/. Prompts carry third-party text (abstracts); on Windows the Codex sandbox cannot be enforced, so each call runs unsandboxed with a tool-forbidding preamble — use --llm api if that is unacceptable. Undermind (Stage 1) and web search (Stage 4d) are agent-only and not available in this runner; the keyless free index search (Stage 4e) runs by default (--no-freesearch to skip). Use it for unattended runs or when no agent is driving. It is the fallback, not the default.

Undermind (Stage 1)

Undermind runs through its official MCP server (https://mcp.undermind.ai/mcp, OAuth on your own account) rather than a browser: the agent launches a deep search from the extracted brief, polls it, pages the result list, and saves the pages. undermind_harvest.py copies the tool results out of the session transcript verbatim (Claude Code persists large tool results to a file it names in the result; Codex keeps them in its rollout log), and undermind_ingest.py parses the [Key] Title (Year) blocks, fills missing DOIs (Crossref) and abstracts (OpenAlex), and writes stage1_undermind.json (+ .bib). The ingest also accepts RIS/BibTeX exports from the web app and plain JSON arrays.

Setup: Claude Code claude mcp add --transport http undermind https://mcp.undermind.ai/mcp then claude mcp login undermind (interactive terminal); Codex [mcp_servers.undermind] in ~/.codex/config.toml. The pipeline stores no Undermind credentials.

If the MCP tools are absent or the search fails, the stage degrades gracefully: print UNDERMIND_DEFERRED, write [] to stage1_undermind.json, and continue on the other channels (Gemini Deep Research, SearchAPI Google Scholar, and the keyless web and free-index searches); the brief stays in undermind_brief.txt for a manual run in the web app (Export → RIS → ingest the file). Uncheck Undermind in the GUI or use --quick to leave it out. The autonomous orchestrator.py has no MCP client and never runs this stage. Recipe and troubleshooting: undermind-search/SKILL.md.

Scholar Labs (Stage 2)

Google Scholar Labs is opt-in (pass --scholarlabs; off by default because Google rate-limits its Cite/BibTeX export under automation, so it frequently defers — Undermind and Deep Research are the dependable deep-search channels). When enabled it runs from the Scholar Labs question Stage 0 writes to scholarlabs_query.txt — a single detailed research question, which is a different input from the Undermind brief. The driver (scholarlabs-search/scripts/scholarlabs_search.py) drives Playwright: it signs in to Google with a persistent profile (~/.scholar-profile), submits the question, waits for the result cards, and reads each result's citation through the standard Scholar Cite → BibTeX export; scholarlabs_ingest.py parses and enriches them into stage2_scholarlabs.json (+ .bib). It returns ~10 highly relevant papers that complement Undermind's broader set. First-time setup seeds the Google session with python scholarlabs-search/scripts/scholarlabs_search.py --login (sign in by hand, including any 2FA).

This stage runs off-screen, not headless. Google serves headless/automated Scholar an "unusual traffic" CAPTCHA, but a real headed Chrome passes — so the orchestrator runs it with --hidden: a genuine headed window positioned far off-screen, reusing the seeded session, so the stage works without anything appearing in front of you. If the session lapses and a 2FA wall appears (which an off-screen window can't clear), the stage defers with a desktop alert telling you to re-run --login (visible) to re-authorize. A missing sign-in or CAPTCHA also degrades gracefully (SCHOLARLABS_DEFERRED, empty results) and the pipeline continues. Disable it with --skip scholarlabs. See scholarlabs-search/SKILL.md.

Input parsing

scripts/manuscript_parser.py is bundled and ships with the skill, so a fresh clone works with no dependency beyond python-docx:

• .docx — the body is walked in document order so paragraphs and tables are captured, and tracked-change text is included (technique adopted from the word-docx skill's extract_text). Footnotes and endnotes are read directly from the document XML so they are never lost.
• .tex — title, abstract, sections, and \footnote{} content are extracted; the bibliography and \cite/\ref keys are stripped so they do not pollute the topic profile. \input/\include targets are inlined.

Short inputs such as a bare abstract are handled: the text becomes a single block and the extractor infers the framing.

API keys

Stored in ~/.lit-review-pipeline.env (auto-loaded; template in lit-review-pipeline.env.example). Real environment variables take precedence.

The autonomous runner's Codex path takes environment variables rather than keys: LITREVIEW_LLM (auto | codex | api), LITREVIEW_CODEX_BIN, LITREVIEW_CODEX_STRONG / _STRONG_EFFORT (default gpt-5.6-sol / xhigh), LITREVIEW_CODEX_WORKER / _WORKER_EFFORT (default gpt-5.6-luna / high), LITREVIEW_CODEX_ESCALATE / _ESCALATE_EFFORT (re-judges low-confidence dedup pairs; default gpt-5.6-terra / high, none disables), LITREVIEW_CODEX_ARGS (extra codex exec args, e.g. -c service_tier=fast), LITREVIEW_CODEX_CONCURRENCY (default 4), and LITREVIEW_CODEX_TIMEOUT (seconds per call, default 900). Undermind needs no key: it runs through the MCP server.

CLI flags

Output

{output-dir}/
├── search_plan.json / .md          # research question, Undermind brief, queries, themes
├── scholar_queries.json            # the Google Scholar query list
├── undermind_brief.txt             # the Undermind brief
├── undermind_page_<k>.txt          # raw Undermind MCP pages (agent flow; harvested from the transcript)
├── scholarlabs_query.txt           # the Scholar Labs research question
├── stage1_undermind.json / .bib    # Undermind results (enriched)
├── stage2_scholarlabs.json / .bib  # Scholar Labs results (enriched)
├── stage2b_deepresearch.json / .bib # Gemini Deep Research results
├── stage4a_scholar.json / .ris
├── stage5_merged.json / .ris       # deduplicated, verified master list
├── stage5_merged_unverified.json   # papers dropped by verification (audit trail)
├── stage6_screened.json/.xlsx/.ris        # ALL screened papers, scored & ranked (no score filter)
├── stage6_screened.bib                     # screened papers (BibTeX), score >= 4 only
├── stage6_filtered.json/.xlsx              # score >= 4 subset (the shortlist)
├── dedup_log.json / dedup_report.md / verification.log
├── _codex/                         # codex exec prompts, replies, and logs (autonomous Codex path)
└── pipeline.log

Skill structure

lit-review-orchestrator/
├── SKILL.md
├── README.md                         # GitHub landing page
├── AGENTS.md                         # Codex repository guidance
├── CLAUDE.md                         # Claude Code repository guidance
├── agents/openai.yaml                # Codex UI metadata and explicit invocation policy
├── docs/
│   ├── codex.md                      # Codex model routing and subagent rules
│   └── claude-code.md                # Claude Code model routing and tool notes
├── requirements.txt
├── lit-review-pipeline.env.example
├── scripts/
│   ├── orchestrator.py            # this controller
│   ├── codex_seam.py              # codex exec seam filler for the autonomous runner
│   ├── lit_review_gui.py          # interactive settings dialog (GUI front door)
│   ├── manuscript_parser.py       # bundled .docx/.tex parser
│   └── extract_search_plan.py     # Stage 0 extraction
├── undermind-search/              # Stage 1 (MCP transcript harvest + ingest)
├── scholarlabs-search/            # Stage 2 (Playwright driver + ingest)
├── deepresearch-search/           # Stage 2b (Gemini Deep Research API)
├── supplementary-search/          # Google Scholar (Stage 4a) + opt-in sources (4b)
├── websearch-search/              # Stage 4d (keyless agent web search)
├── freesearch-search/             # Stage 4e (keyless OpenAlex/Crossref/Semantic Scholar)
├── lit-dedup/                     # Stage 5 + 5b verification
├── lit-screen/                    # Stage 6
└── examples/sample_manuscript.tex

Operational notes

• Run in the foreground; the dashboard streams stage status to the console and pipeline.log.
• Stages 5 (dedup) and 6 (screen) can be re-run standalone after editing any stage input file.
• Citation chaining needs DOI-bearing seeds, so it is most useful once Undermind enrichment is active.
• Stage 0 failure aborts the run (the pipeline cannot proceed without a search plan).
• The SSRN, HeinOnline, and forthcoming sources are Google Scholar searches with a site:/source: filter — subsets of Stage 4a — so they mainly force those venues to surface rather than adding a new index. NBER (nber.org API) and citation chaining (Semantic Scholar) hit independent indexes and add genuine coverage.
• --quick runs only the SearchAPI Google Scholar channel (no browser, no login), useful for a fast pass or when the Undermind MCP server / Scholar Labs login are unavailable.