Undermind Search

Undermind Search (Stage 1)

Takes the Undermind brief that Stage 0 extracts and returns the papers Undermind's deep search finds, enriched and saved as <stem>.json + <stem>.bib for dedup and screening.

The stage is agent-driven end to end. There is no subprocess driver: the host agent calls the Undermind MCP tools, saves what they return, and one script normalizes it.

• The agent — launch_deep_search → poll inspect_deep_searches → page the results.
• scripts/undermind_harvest.py — copies the paged tool results out of the session transcript byte-for-byte into OUT/undermind_page_<k>.txt (Claude Code ~/.claude/projects/<slug>/<session>.jsonl, following its <persisted-output> pointers; Codex ~/.codex/sessions/**/rollout-*.jsonl).
• scripts/undermind_ingest.py — parses those files (or any RIS/BibTeX export), fills missing DOIs (Crossref) and abstracts/journals (OpenAlex), and writes the pipeline JSON. Importable, and runnable standalone.

Prerequisite: the Undermind MCP server

Undermind's official MCP endpoint is https://mcp.undermind.ai/mcp (OAuth, your own Undermind account).

• Claude Code: claude mcp add --transport http undermind https://mcp.undermind.ai/mcp then claude mcp login undermind in an interactive terminal. A session that was already open when you logged in only exposes mcp__undermind__authenticate; call it once, approve in the browser, and the full tool set swaps in.
• Codex: in ~/.codex/config.toml:

  [mcp_servers.undermind]
  url = "https://mcp.undermind.ai/mcp"
  [mcp_servers.undermind.oauth]
  client_id = "codex"
  

  Codex completes the OAuth flow on first use.

Call get_orientation once per conversation before the other tools.

The flow

brief ─► create_workspace (or reuse) ─► launch_deep_search(goal=brief)
      ─► poll inspect_deep_searches(status_only) every ~60 s (2–5 min)
      ─► inspect_deep_searches(papers_only, detail_level="full", limit=50, offset=0/50/…)
      ─► undermind_harvest.py --latest --match <name> --out-dir OUT   (pages → OUT/undermind_page_<k>.txt)
      ─► undermind_ingest.py --input OUT/undermind_page_*.txt -o OUT/stage1_undermind.json

Step by step, with OUT the run's output directory:

1. list_workspaces → reuse a workspace for this manuscript, or create_workspace(title=<document stem>). Keep the workspace_id.
2. launch_deep_search(workspace_id, goal=<contents of OUT/undermind_brief.txt>, name=<short label>). The tool returns immediately. Launch it first, then start the other channels while it runs. The brief goes in verbatim: the goal is meant to be a self-contained paragraph, which is exactly what Stage 0 writes (no keyword lists, no journal names). The MCP search asks no clarifying questions; the plan's undermind_clarifications field is informational only (a holdover from the browser flow).
3. Poll inspect_deep_searches(names=[<name>], status_only=true) from a background Monitor, not a foreground wait. A rate_limited result means wait and retry; Undermind runs only a few deep searches concurrently.
4. On completion, page the papers: inspect_deep_searches(names=[<name>], papers_only=true, detail_level="full", limit=50, offset=0), then offset=50, 100, … until a page comes back short. Do not retype or summarize anything.
5. Harvest the pages from the transcript, then ingest:

   python undermind-search/scripts/undermind_harvest.py --latest --match "<name>" --out-dir OUT
   python undermind-search/scripts/undermind_ingest.py --input OUT/undermind_page_*.txt \
       -o OUT/stage1_undermind.json
   

   The harvester scans the newest transcript under the Claude Code projects dir (CLAUDE_CONFIG_DIR or ~/.claude) and ~/.codex/sessions, keeps every distinct tool result that contains [Key] Title (Year) blocks and whose header (the search-name line above the paper list) contains the --match text, and writes them as undermind_page_01.txt, _02, …; without --match, when the transcript holds several searches, only the latest search's pages are kept and the others are named. Pages whose showing a-b of n ranges leave a gap raise a WARNING naming the missing offsets. (--transcript PATH to name the file; --min-records N to drop short fragments). Falling back to saving a page by hand as .txt or as the {"result": "..."} JSON also works: the ingest reads both.

Optional add-ons before ingest, for the same workspace: search_papers with search_type="citations" / "references" seeded on the top cite keys widens coverage; save its output as another page file and ingest it together.

Input formats the ingest accepts

Auto-detected from content:

Several --input files are concatenated and deduplicated by cite key, DOI, or normalized title, so paginated dumps just go in together.

CLI

undermind_harvest.py:

undermind_ingest.py:

Graceful degradation

If the MCP tools are unavailable (server not added, not authenticated, or a Codex/Claude session without MCP), the agent prints UNDERMIND_DEFERRED, writes [] to stage1_undermind.json, and continues; the ingest does the same when no input file exists or nothing parses. The brief is always available in undermind_brief.txt for a manual run in the web app (Export → RIS, then ingest the file). The autonomous orchestrator.py never runs this stage: it has no MCP client.

Output schema

Same as the other stages, with source: "undermind":

{"title": "...", "authors": "A, B", "year": "2024",
 "doi": "https://doi.org/10.x/y", "abstract": "...", "journal": "...",
 "url": "...", "source": "undermind", "verified": true,
 "citations": 0, "open_access": false}

Troubleshooting

• Only mcp__undermind__authenticate is available — the session predates the login. Call it, approve in the browser, and the tools appear.
• claude mcp list says "Needs authentication" although login worked — a stale mcp-needs-auth-cache.json in the Claude config dir; harmless to delete.
• rate_limited — Undermind caps concurrent deep searches; wait and retry.
• The harvester finds no pages — the tool results have not been flushed to the transcript yet, or the newest transcript belongs to another session; wait a moment and retry, or pass --transcript with the session file. --match is case-insensitive and is tested against the page header (the search-name line), never against paper text.
• A page parses to zero records — the tool output format drifted. Run --selftest, compare the saved page with the header pattern in parse_mcp_text, and adjust; the JSON-array input is the format-independent fallback.