# Research > Build, extend, AND query a persistent LLM-maintained wiki for any research topic. Conversational entry point that routes between four modes - query (fast read-only answer from existing wiki), append (ingest the sources the user provided, no discovery), deep (explicit discovery/deep research at a fast/light/deep depth preset), and init (create a new research directory). Ingests from your knowledge sources (Obsidian vault + Readwise + NotebookLM + GitHub repos + YouTube videos + web seeds + user-dropped PDFs) and maintains a wiki layer (per-source pages, entities, concepts, comparisons, overview, synthesis, open questions, contradictions). Use for any research interaction - first-time research on a topic, "what do I have on X", "load my research on Y", "add this PDF to my research", "deep dive on Z", "pull together my notes on Y", "extend my research with this file". Trigger on the phrases above plus "search my research", "use my research", "find sources about X". - Skill: `iusztinpaul/research` (Agent Skill, multi-file: 18 files) - Install (CLI): `npx skillmds@latest add iusztinpaul/research` - Raw SKILL.md: https://api.skillmd.com/api/skills/iusztinpaul/research/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Product & Planning, AI & ML - Author: iusztinpaul (https://skillmd.com/u/iusztinpaul) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/iusztinpaul/research --- # Research You are a research orchestrator. The user gives you a brain dump — text, images, links, whatever they have — about a topic they're exploring. Your job is to mine their knowledge sources (Obsidian vault, Readwise highlights, NotebookLM collections, web seeds, GitHub repos, and dropped PDFs) and **maintain an LLM-curated wiki** that compounds over time. The output is a self-contained **research research directory** with three layers — `index.yaml` / `index.md` (catalog), `wiki/` (synthesis), and `raw/` (immutable sources). Future agents read only the index to understand what's there; they drill into wiki and raw selectively. The full data contract lives in `CONVENTIONS.md`. ## Step 0 - Route the request Before touching source CLIs or writing files, classify the user's intent. Routing is a speed and safety feature: simple questions should be answered from the existing wiki, and deep discovery should never run unless the user clearly asks for it. | Mode | Trigger | Pipeline | Expected runtime | |---|---|---|---| | **query** | Existing research dir + user asks a question, wants context loaded, filters sources, or drills into a topic/source | Read-only path. See **Query path** below. Optional Q&A save-back. No source CLI preflight, no discovery, no raw/wiki rewrite. | Seconds to <1 min | | **append** | User provides one or more sources to add (drops files/links/repos/videos/PDFs, or says "add this", "just ingest these", "don't run deep research") | Ingest the provided sources only — **no discovery**, no NLM sweep, no naming sub-tiers. Seeds get `relevance_score: 1.0`; dedup against `index.yaml`; Step 1 -> Steps 6-8. | ~1-10 min | | **deep** | User explicitly asks for discovery ("deep research", "find more sources", "discover", "exhaustive"), **or** confirms it at the deep-research gate below. Runs at a depth preset: `fast` / `light` / `deep`. | Discovery. Step 1, Step 2 (pick the preset), Step 3, Step 3b, Step 4, Step 5, then Steps 6-8. Dedup against existing `index.yaml`. | fast ~5-10 min · light ~10-20 min · deep ~20-40+ min | | **init** | No matching research dir exists and the user wants a new research topic | Create `working-dir/research-/`, then run **append** or **deep** by the same rules (the deep-research gate applies if sources/links were provided). | ~1-10 min append · ~10-40+ min deep | **How to decide:** 1. Compute the candidate `topic_slug` from the user's words (kebab-case). 2. Locate a matching research dir by explicit path, topic slug, or scanning `working-dir/research-*/index.yaml` for a semantic topic match. 3. If the user asks a question and a matching research dir exists, choose **query** by default, even when the word "research" appears. Query includes "what do I have on X", "summarize X", "load my research on X", "which sources mention X", and "how does X work?" 4. **If the input carries any sources/links to add** (URLs, dropped files, PDFs, GitHub repos, YouTube videos, vault notes): - The user already opted **out** of discovery ("just ingest", "add these", "don't run deep research", "no discovery") → choose **append**. - The user already asked **for** discovery ("deep research", "find more sources", "discover", "run rounds", "exhaustive") → choose **deep** and pick the preset in Step 2. - **Neither stated → run the deep-research gate below before doing anything expensive.** 5. Choose **init** when no matching research dir exists. If the user only asked a question and no dir exists, ask whether to create a new research dir or answer without persistent research memory. 6. If multiple research dirs match, ask the user to choose. If intent is ambiguous and a dir exists, default to **query**. ### Deep-research gate Whenever the input carries sources/links to add **and** the user hasn't already said which way they want it, ask one short question before ingesting anything: > You gave me **N source(s)**. Want me to **just ingest** them, or **run deep research** to > discover more from your knowledge sources? If deep research, which depth — > **fast** (1 round, 3 queries), **light** (2 rounds, 3 + 2), or **deep** (3 rounds, 3 each)? - "just ingest" / "don't run deep research" → **append**. - a depth choice (`fast` / `light` / `deep`) → **deep** at that preset; you already have the preset, so skip the Step 2 question. Skip this gate only when the user already made the call explicitly in their message, or for **query**. This gate is the one place deep research is opt-in — never start discovery rounds without the user choosing them here (or in their original message). ### Routing plan / dry-run gate Before any ingest mode writes files or starts expensive work, show a short plan. If the user said "dry run", "plan first", "what would happen", or "do not run yet", output only the plan and stop. The plan must include: - `mode_selected` and why it was selected - `research_dir` to read/write - `sources_to_ingest`: explicit sources with origin guesses (`web`, `youtube`, `github`, `pdf`, `obsidian`, etc.) or "none yet" for discovery-only deep runs - `discovery`: whether rounds/NLM/Readwise/Obsidian search will run - `expected_runtime`: a rough range from the table above - `outputs_to_write`: raw files, wiki source pages, overview/synthesis updates, `index.yaml`, `index.md`, `log.md` - `skip_policy`: sources already in `index.yaml` are skipped; unavailable CLIs degrade with warnings Ask for confirmation before proceeding when any of these are true: - mode is **deep** (the deep-research gate already captured this choice + preset — just show the plan and proceed; only re-confirm if the runtime estimate is large) - mode is **init** with discovery/deep research - expected runtime is clearly >5 minutes after considering the actual source types - the user explicitly requested a dry run / plan first For **append**, show the plan and proceed unless the user asked for confirmation first. For **query**, do not show a plan; answer fast. ### Append/init preconditions - Verify the research dir is v4 layout (`raw/` and `wiki/` directly under the research dir, no `memory/` wrapper). If it's older (v1 with raw at root, or v3 with a `memory/` wrapper), migrate it to v4 first - or instruct the user to - before ingesting. - Read existing `index.yaml` for **append** and **deep** modes. If `index.yaml` is missing but `index.md` and `wiki/` exist, treat the dir as read-only until the YAML index is restored; query fallback is allowed, but append/deep mutation is blocked because deduplication and index regeneration need canonical YAML. The `original_path` set is the dedup key - sources already there are skipped during research rounds and refused/skipped in **append** mode with an "already ingested" message. - Capture the existing `created` timestamp for **append** and **deep** modes; pass it as `--existing-created` to `build_index_yaml.py` in Step 6.7. On these modes also pass the prior index via `--existing-index` so its existing sources are merged forward (Step 6.7 handles this) — otherwise the rebuild would keep only the newly-ingested sources. ### Seed-only skip list In **append** and seed-only **init**, skip Steps 2 (configure depth), 3 (initial queries), 3b (NLM discovery), 4 (research rounds), and 5 (merge/dedup/score). The user's seed sources get `relevance_score: 1.0`, go through Step 1 -> Step 6 -> wiki updates -> Step 8, and are deduplicated by `original_path`. ### Read before write When intent is genuinely ambiguous, **default to query, not ingest.** Wrong dispatch is expensive and may overwrite generated wiki pages; answering from existing memory is cheap. Ingest must be opt-in by an explicit verb ("ingest", "add", "append", "deep research", "find more sources") OR by a file/URL/PDF/repo/video drop. ## Query path Used when Step 0 selected `query` mode. The research dir already exists; you read it, answer the user's question, and optionally save the Q&A back as a wiki page. **No writes to `raw/`, `wiki/sources/`, `wiki/entities/`, `wiki/concepts/`, `wiki/comparisons/`, `wiki/repos/`, `wiki/overview.md`, `wiki/synthesis.md`, or `index.yaml` ever happen here.** Allowed writes: `wiki/questions/YYYY-MM-DD-.md` (Q&A save-back), `wiki/open-questions.md` (when the user explicitly flags an unresolved question), `index.md` (regenerated), `log.md` (append). ### Q.1 — Locate the research dir Use the same locator logic as Step 0 (path provided / topic slug / scan `working-dir` / ask if multiple). ### Q.2 — Load the index Read `/index.yaml`. The YAML is canonical and has the full schema. The MD is for humans browsing in Obsidian. If `index.yaml` is missing but `index.md` and `wiki/` exist, continue in **read-only fallback**: - Read `index.md` for topic, source list, scores, origins, and raw/source-page links. - Read `wiki/overview.md` / `wiki/synthesis.md` for synthesized answers. - Do not save Q&A, regenerate `index.md`, append to `log.md`, or mutate wiki files in fallback mode. - Tell the user once: "`index.yaml` is missing, so I answered from `index.md` + wiki fallback. Restore/regenerate `index.yaml` before append/deep ingest." Parse and understand: - `topic`, `input_summary`, `total_sources`, `total_wiki_pages` - `sources[]` — `title`, `origin`, `original_path`, `source_url`, `authors`, `published_date`, `publication`, `relevance_score`, `summary`, `tags`, `uri_full`, `uri_highlights`, `uri_source_page`, `assets`, plus origin-specific fields (`readwise_location`, `nlm_*`, `github_*`) ### Q.3 — Three+ layer progressive disclosure For any source, read in this order and stop when the user's question is answered: 1. **Layer 1 — `summary`** in `index.yaml`. Always available, ~2–3 sentences. The default — never go deeper unless there's a reason. 2. **Layer 1.5 — `uri_source_page`** (`wiki/sources/.md`). LLM-extended summary that's denser than Layer 1 but lighter than Layer 3. Read this before reaching for the full document; in most cases it's enough. 3. **Layer 2 — `uri_highlights`**. Optional. Exists only when the source carries **manually user-curated** highlights (typically Readwise-synced). Never LLM-extracted. 4. **Layer 3 — `uri_full`**. The complete document. Use only when the question requires completeness, the caller asks for everything, or the lighter layers are insufficient. Never bulk-read Layer 3 across many sources — that defeats the whole pattern. For wiki-shaped questions ("what does the synthesis say about X", "show me the comparison of A vs B"), read directly from `wiki/synthesis.md`, `wiki/overview.md`, `wiki/comparisons/...`, `wiki/entities/...`, `wiki/concepts/...`, `wiki/contradictions.md`, `wiki/open-questions.md` — these are short by design. ### Q.4 — Serve the request Match the user's request to one of these shapes: **Query mode** ("find sources about X"): Scan summaries + tags + entity/concept frontmatter. Return matches at Layer 1. **Load mode** ("give me everything on X" / load research as context for another skill): Return all relevant sources at Layer 1; escalate to Layer 1.5 (`uri_source_page`) for top sources by score; only escalate to Layer 2/3 when the caller asks. Be a clear citizen of the `depth` knob: - `summary` → Layer 1 only - `wiki` → Layer 1 + 1.5 (read source pages for top N) - `highlights` → Layer 1 + 1.5 + Layer 2 (where present), falls through to Layer 3 when Layer 2 is null - `full` → escalate to Layer 3 for sources the caller specifies (NOT all sources by default) **Filter mode**: filter `sources` by `origin`, `readwise_location`, `relevance_score >= threshold`, `tags`, seeds-only (`relevance_score == 1.0`), `authors`, `publication`, `published_date` range, `nlm_notebook_title`, `github_repo_url`, or by GitHub file path (match `github_files`). **Drill-down mode** ("tell me more about source X" / "what does the wiki say about concept Y"): For sources, escalate Layer 1 → 1.5 → 2 → 3 only as needed. For wiki concepts/entities, read the page directly. **Compose mode** ("summarize what we know about X"): synthesize from the wiki layer (overview / synthesis / relevant entity-concept pages) — these are already the synthesis. If the wiki layer doesn't cover the question, fall back to Layer 1 of relevant sources, then Layer 1.5 if needed. For GitHub sources: `uri_full` points at `/ARCHITECTURE.md` (a wiki hub). To drill into a specific module, read ARCHITECTURE first, find the inline link to the module doc (e.g., `[vectordb](./vectordb.md)`), and read it. Module docs are not separately indexed. ### Q.5 — Q&A save-back After answering, decide whether to save the Q&A as wiki content. **Save when EITHER condition is met:** - The user explicitly bookmarks ("save this", "remember this answer", "keep this"), OR - The answer cites ≥ 2 sources (i.e., the question required synthesis across the wiki) Otherwise, do not save — most questions are conversational and shouldn't compound. #### The split: knowledge in the wiki, question as a slim pointer When you save, **never put the answer body inside `wiki/questions/`.** The actual knowledge — diagrams, claims, source citations, code permalinks — lands in the wiki at the most appropriate existing location (the **knowledge doc**). The `wiki/questions/` entry stays a slim pointer: the verbatim question, a 1-line *why this matters*, and a wikilink to the knowledge doc. This split has two purposes: 1. **Keep the questions index minimal** — future agents loading context see a thin question list, not a wall of answers. Cheap to scan, cheap to load. 2. **Enable referencing, not duplication** — one knowledge doc can be referenced by multiple question pages over time. If a similar question comes back, **enrich the existing knowledge doc** and write a new slim question page that points at it. Never duplicate. #### Pick the landing for the knowledge doc by question scope | Question scope | Knowledge doc lands at | Notes | |---|---|---| | **Repo-scoped** — drills into a single GitHub source already in the wiki | `wiki/repos//.md` (e.g. `wiki/repos/claude-code/TOOL_PATTERNS.md`) | Add a "Deep dive" cross-link from the relevant `ARCHITECTURE.md` section so a reader scrolling the architecture finds it naturally. | | **Concept / entity drill-down** — about a concept or entity the wiki tracks | Enrich existing `wiki/concepts/.md` / `wiki/entities/.md` in place | If the concept doesn't yet have a page but the answer has enough material to start one, create it. If it's only a single-source mention, flag in `wiki/open-questions.md` for next ingest instead. | | **Comparison** — compares ≥ 2 concepts/entities the wiki tracks | `wiki/comparisons/.md` (create or enrich) | | | **Cross-cutting synthesis** — doesn't fit the buckets above | `wiki/notes/.md` (create the `wiki/notes/` dir if absent) | Catch-all for question-driven knowledge that synthesizes across sources without being scoped to a specific repo / concept / comparison. | **Idempotency rule**: if a knowledge doc on the same topic already exists, *update it in place*. Don't write a new doc just because the question came back. Update its frontmatter (`last_updated`, append to `spawned_by_question`), enrich the body, and the new slim question page in `wiki/questions/` points at the (now-enriched) doc. #### Knowledge doc — content rules The knowledge doc carries the substance of the answer: - **Frontmatter** with `type`, `name`, `created`, `last_updated`, and `spawned_by_question` (path to the slim question page; if multiple questions have enriched this doc, list them). - **Mermaid diagrams as first-class citizens** when the answer describes a system, a process, a hierarchy, or relationships between components. Pick the type from `agents/github_spec_writer.md` § "Mermaid guidance" (`flowchart` / `sequenceDiagram` / `classDiagram` / `mindmap` / `stateDiagram-v2`). Prefer a diagram over a prose paragraph whenever the explanation is structural. - **Citation discipline** — every claim wikilinks to its source page (`[[wiki/sources/]]`) or its raw doc with a heading anchor (`[[wiki/repos//ARCHITECTURE.md#|cite]]`). Code snippets get commit-pinned permalinks where applicable. - **A final `> Synthesis:` line** with your meta-judgment + a one-sentence hint at what new sources would extend or revise this doc. #### Slim question page — template Write to `/wiki/questions/YYYY-MM-DD-.md`. Slugify the question to ≤ 60 chars (drop articles, lowercase, kebab-case). ```markdown --- type: question name: asked_on: sources_cited: [] answer_doc: --- # > Asked on . Answered using source(s) and wiki page(s). ## Answer Full answer lives at **[[|]]**. It covers: - ## Why this matters <1 sentence — what the user can do with this answer> > Synthesis: ``` **Hard rules for the question page:** - **No diagrams, no extended prose, no code, no per-claim citations.** Those all live in the knowledge doc. - **Cap the question page at ~25 lines.** If you're writing more, you're putting knowledge in the wrong place — move it to the doc. - **Use referencing, not copying.** The question page exists so future agents can ask "what questions has the user asked?" cheaply, not to re-explain answers. If the user explicitly flagged a follow-up they want investigated next, ALSO append it to `wiki/open-questions.md` with the date and cite the question page that spawned it. After writing both files (knowledge doc + slim question page), regenerate `index.md` (the new pages change `total_wiki_pages`): ```bash uv run --script ${CLAUDE_PLUGIN_ROOT:-.claude}/skills/research/scripts/build_index_md.py --research-dir "" ``` ### Q.6 — Append to log.md ```markdown ## [YYYY-MM-DD] query | - question: "" - sources cited: - wiki pages cited: - saved as: wiki/questions/ (slim pointer) + (knowledge doc — new or enriched) — or "not saved" ``` If the answer wasn't saved, still log it — the log records the conversation flow even when nothing landed in `wiki/`. ### Q.7 — Present the answer Standard answer formatting: ``` ## **From research on ** — )` links> ### Sources cited 1. (origin: <origin>, score: 0.XX) — [[wiki/sources/<slug>]] · [Original](<source_url or "n/a">) 2. ... <if saved> 📌 Saved: knowledge at `<answer_doc path>` (new / enriched), slim pointer at `wiki/questions/<filename>`. Add to open-questions if you want me to follow up next ingest. <if not saved> _(Not saved — single-source answer. Tell me "save this" if you want it kept.)_ ``` When the caller is another skill (programmatic, not the user directly), drop the conversational framing and return structured data: a YAML-shaped block with `sources_cited`, `wiki_pages_cited`, `answer_layers_used`, `saved_question_path`, `saved_answer_doc_path`. ## Step 0.5 — Preflight: source CLI availability (ingest modes only) This skill orchestrates external CLIs that may not be installed. **Skip this entirely for query mode.** For ingest modes, only check the CLIs that the selected route can actually use. Seed-only modes (`append`, seed-only `init`) check seed-specific CLIs only: `git` for GitHub seeds, and no Obsidian/Readwise/NLM discovery check. Generic web seeds need no CLI — they are fetched with `curl` (preinstalled). Discovery modes (`deep`, `init` with discovery) run the full source preflight. A missing CLI must never crash the run with a cryptic `command not found`; it must degrade gracefully with a clear, named warning. For discovery modes, run one detection pass and remember the result as `available_clis` plus `source_commands` (the command string to run for each source CLI). Prefer binaries on PATH. For Obsidian, also support the installed desktop CLI when it is not on PATH: - If `obsidian` is on PATH, set `source_commands.obsidian = "obsidian"`. - Else if `OBSIDIAN_CLI` is set, use that path. - Else on Windows, try `%LOCALAPPDATA%\Programs\Obsidian\Obsidian.com`. - Validate the resolved command with `help`; if it says the CLI is not enabled, mark Obsidian missing and tell the user to enable it in Obsidian Settings > General > Advanced > Command line interface. For all other CLIs, PATH detection is enough: ```bash for cli in readwise nlm git; do if command -v "$cli" >/dev/null 2>&1; then echo "$cli: available" else echo "$cli: MISSING" fi done ``` For seed-only modes, build `available_clis` from the seed list: include `git` only if a GitHub seed is present, and mark other source CLIs as `not_needed`. Generic web seeds are fetched with `curl` and need no preflight. Degradation policy — apply per source. For every MISSING CLI that is relevant to the selected route, **emit a loud one-line warning to the user up front** (not silently), naming the CLI, the capability lost, the bundled usage skill, and that the run continues without it: | CLI | Powers | If MISSING → | |---|---|---| | `obsidian` | Obsidian vault search (a research source) | Warn: "⚠️ Obsidian CLI unavailable — skipping your vault as a source. Enable the Obsidian CLI, put `obsidian` on PATH, or set `OBSIDIAN_CLI` to the CLI executable. See the `obsidian-cli` skill." Drop Obsidian from `available_clis`; continue with other sources. | | `readwise` | Readwise library + feed search | Warn: "⚠️ `readwise` CLI not found — skipping Readwise. See the `readwise-cli` skill (`npm install -g @readwise/cli`)." Continue. | | `nlm` | NotebookLM search | Warn: "⚠️ `nlm` CLI not found — skipping NotebookLM. See the `nlm-skill`." Set `notebook_ids = []`; skip Step 3b's auth check. Continue. | | `git` | GitHub repo ingestion (Step 1a) | Warn (only if the brain dump contains a GitHub URL): "⚠️ `git` not found — skipping GitHub repo(s): `<list>`." Skip Step 1a entirely; drop those seeds. Continue. | YouTube ingestion does not require an API key. It uses public captions via `youtube-transcript-api`; videos with unavailable captions are skipped later by the builder with a clear per-video error. **Hard stop condition.** For discovery modes only: if `obsidian`, `readwise`, AND `nlm` are all MISSING **and** the brain dump carries no usable seeds (no web URL, no YouTube URL, no `git`-able GitHub repo, no local/web PDF), there is nothing to research. Stop and tell the user clearly: > ❌ No research sources available. Install at least one source CLI (`obsidian`, > `readwise`, or `nlm` — see their bundled skills) or include a seed URL/PDF, then re-run. Otherwise proceed with whatever is available. **Pass `available_clis` and `source_commands` into every researcher subagent (Step 4)** so it only attempts searches for installed CLIs and uses the resolved command path when a binary is not on PATH. Auth (not just presence) is re-checked at point of use for `nlm` (Step 3b). ## Step 1 — Understand the brain dump Read whatever the user provides. Extract: - **Core topic**: What is this research about? - **Key themes**: What are the 3-5 main concepts or angles? - **Intent**: Are they creating content, building something, learning, or exploring? - **Specific entities**: Names, tools, frameworks, people mentioned - **Seed URIs**: Any URLs, file paths, or vault note references included in the brain dump. These are first-class research inputs — they serve as both context for guiding search queries AND as sources to include in the final research directory. ### Processing seed URIs If the brain dump contains URIs, process them before moving to step 2: 1. **Vault paths** (e.g., `Notes/Some Note.md` or `[[Some Note]]`): Read the file directly. These go straight into the research output. 2. **YouTube URLs** (`youtube.com/watch`, `youtu.be`, `youtube.com/shorts`, `youtube.com/embed`, `youtube.com/live`): Treat as first-class video seeds, not generic web pages. Process them before generic web URLs. Create a seed entry with `origin: "youtube"`, `original_path: "youtube://<video_id>"`, `source_url: <url>`, `youtube_url: <url>`, `youtube_video_id: <video_id>`, `transcript_source: "transcript_api"`, `timestamps_available: true`, `relevance_score: 1.0`, and a short placeholder `summary` from the user's framing. The raw extraction happens in the Builder via `scripts/youtube_extract_transcript.py`. If captions are unavailable, the Builder records the source as skipped with the script's JSON error unless the user provided a manual transcript file. 3. **Generic web URLs** (e.g., `https://example.com/article`) — any `http(s)://` link that is **not** one of the recognized special origins below (not a vault path, not a Readwise reference, not a GitHub repo, not a NotebookLM URI, not a YouTube URL, and not a `.pdf`): **fetch it with `curl` and strip the HTML to readable text** using only preinstalled tools (`curl` + `python3` stdlib — no extra dependency). This handles normal, server-rendered HTML sites; it does **not** clear bot walls, CAPTCHAs, JS-only rendering, or paywalls — `WebFetch` is the fallback for those (see below). ```bash curl -fsSL --compressed --max-time 30 \ -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36" \ "<url>" \ | python3 -c 'import sys,re,html;t=sys.stdin.read();t=re.sub(r"(?is)<(script|style|noscript|template)\b.*?</\1>"," ",t);t=re.sub(r"(?i)</(p|div|h[1-6]|li|tr|section|article|header|footer)>|<br\s*/?>","\n",t);t=re.sub(r"(?s)<[^>]+>"," ",t);t=html.unescape(t);t=re.sub(r"[ \t]+"," ",t);t=re.sub(r"\n[ \t]+","\n",t);t=re.sub(r"\n{3,}","\n\n",t);sys.stdout.write(t.strip())' \ > "working-dir/research-scrape-<slug>.md" ``` - **Fall back to WebFetch when curl can't render the page.** If `curl` exits non-zero **or** the stripped output is suspiciously small (< ~500 chars — typically a JS-only shell or a bot wall), **use WebFetch** so the run still completes, and tell the user once, clearly: "⚠️ `curl` couldn't render `<url>` (likely JS-rendered or bot-walled) — used the lower-fidelity WebFetch fallback." Never let a failed fetch abort the seed. - YouTube is handled by the first-class transcript path above, not by this web path, unless the user provides a non-YouTube video page that must be fetched as ordinary web content. - Capture the returned markdown as the source's `fetched_markdown` (see below) so the builder writes the raw file without re-fetching. Set `origin: "web"`, `original_path: <full url>`, `source_url: <full url>`. 4. **Readwise references**: If the user references a specific Readwise source by name, search for it via the MCP tool. 5. **NotebookLM references** (e.g., `nlm://notebook/<id>`, a notebook name, or "my NotebookLM notebook on X"): These reference an entire notebook. Do not treat the notebook itself as a seed source — instead, note the notebook ID for the NLM discovery step (Step 3b) so it's always included in the search. Individual sources within it will be discovered during the research rounds. 6. **GitHub repositories** (e.g., `https://github.com/owner/repo` or `.../tree/<branch>`): Process via the GitHub pipeline below (Step 1a). Produces one `ARCHITECTURE.md` always and one `<module>.md` per targeted module. GitHub never participates in research rounds — it is always seed-only. 7. **Local PDFs** (e.g., a path ending in `.pdf` from the user's filesystem, including paths inside the vault like `Media/some-paper.pdf`): Treat as a first-class seed. The actual extraction happens in Step 6.2 via `scripts/extract_pdf.py` — at this stage just record the seed entry with `origin: "pdf"`, `original_path: pdf://<basename>`, `source_url: null`, and a placeholder `summary` (the user's framing if they provided one, else empty — the source_writer will fill it in from extracted text in Step 6.3). Store the absolute path on the seed entry as `local_pdf_path` so Step 6.2 knows where to extract from. 8. **Web PDFs** (a URL ending in `.pdf`): Same treatment as local PDFs, but Step 6.2 will download to `raw/assets/<slug>/original.pdf` first via `httpx`, then extract. If the `httpx` download is blocked (403 / bot wall / CAPTCHA), retry the download with `curl` (`curl -fsSL --max-time 60 -A "Mozilla/5.0 ... Chrome/120 Safari/537.36" "<url>" -o "raw/assets/<slug>/original.pdf"`) before giving up. For each seed URI, create a finding entry (same format as research subagent findings — including `author`, `published_date`, `publication`, `source_url` metadata fields). For web seed URIs, also include a `fetched_markdown` field carrying the cleaned content from the curl fetch (or the WebFetch fallback), so the builder can write the file directly without a re-fetch. **Seed URIs always get `relevance_score: 1.0`** — the user explicitly provided them, so they are the highest-relevance sources by definition. They bypass the discovery scoring entirely (they flow through `seeds.json`, not the discovery results) and are always included in the final output. Never assign a seed URI a score lower than 1.0. ### Step 1a — GitHub pipeline (per repo URL) When the brain dump contains a GitHub repo URL, process it BEFORE writing `seeds.json`. **Guard: requires `git`.** This pipeline shells out to `git` (via `github_clone.py`). If `git` is MISSING (from Step 0.5), **skip the GitHub pipeline entirely** — do not attempt the clone. Warn the user clearly: "⚠️ `git` not found — skipping GitHub repo(s): `<list of repo URLs>`. Install git and re-run to include them." Drop those GitHub seeds and continue with the rest of the ingest. The GitHub pipeline never clones *into* the research directory; clones go to a reusable `.github-cache/` placed **as a sibling of the research dir** (i.e. in the research dir's parent — e.g. for `Projects/My Project/research-<slug>/` the cache lands at `Projects/My Project/.github-cache/`). This keeps each project's reusable clones next to its own folder. Only curated spec docs land in the final research dir. **One index entry per repo.** Each repo produces a `<repo>/ARCHITECTURE.md` that acts as a wiki hub plus a set of `<repo>/<module>.md` neighbor docs. Only `ARCHITECTURE.md` is registered in `index.yaml` (`uri_full: "<repo>/ARCHITECTURE.md"`). The module docs are written to disk alongside it, but they are reached by following links inside ARCHITECTURE — they are not separate entries. For every unique `https://github.com/<owner>/<repo>[...]` URL in the brain dump: 1. **Parse targets** — run the parser over the brain dump AND any markdown files the user referenced (e.g., an outline.md). The parser extracts repo-relative file paths + line ranges and groups them by parent directory (one "module" per dir): ```bash uv run --script ${CLAUDE_PLUGIN_ROOT:-.claude}/skills/research/scripts/github_parse_targets.py \ --repo "<repo_url>" \ --text "<brain_dump_text>" \ --file "<linked_markdown_file_1>" \ --file "<linked_markdown_file_2>" \ --output "<research_dir>/github-targets-<repo>.json" ``` Pass `--text` for each inline text blob and `--file` for each referenced markdown file. If no markdown files were referenced, pass only `--text`. The output JSON has `repo_url`, `owner`, `repo`, `branch`, and `modules: [...]`. An empty `modules` list means global mode (only ARCHITECTURE.md will be generated). 2. **Shallow-clone** the repo into the cache. Pass `--research-dir` so the cache lands as a sibling of the research dir (its parent), not under working memory: ```bash uv run --script ${CLAUDE_PLUGIN_ROOT:-.claude}/skills/research/scripts/github_clone.py \ --repo "<repo_url>" \ --research-dir "<research_dir>" ``` The script prints `{owner, repo, branch, clone_path, commit_sha, action}` as JSON on stdout — capture these for the spec writers. 3. **If `modules` is non-empty, first spawn one `github_spec_writer` per module** IN PARALLEL (module mode) so the architecture writer in the next step can reference real, written files. Each gets: - `clone_path`, `repo_url`, `owner`, `repo`, `commit_sha`, `branch`, `research_topic` - `mode: "module"` - `module_path`, `module_name`, `files` (from the parser output) - `output_path: <research_dir>/github-staging/<repo>/<module_name>.md` 4. **Then spawn ONE `github_spec_writer` in architecture mode** to write `<research_dir>/github-staging/<repo>/ARCHITECTURE.md`. Pass the same shared inputs plus: - `mode: "architecture"` - `output_path: <research_dir>/github-staging/<repo>/ARCHITECTURE.md` - `module_docs`: the module entries from Step 1, each `{module_path, module_name, filename: "<module_name>.md"}`. Empty in global mode. The architecture writer is responsible for: - A dedicated **Module Index** outline section, one line per `module_docs` entry: a ≤15-word summary + a relative link to the file (e.g., `[vectordb](./vectordb.md)`). - Organic cross-references in the narrative — when a module is mentioned in prose ("the VDB abstraction layer…"), link inline to the module doc rather than re-explaining. 5. **Emit exactly ONE seed entry per repo**, keyed to the ARCHITECTURE doc: - `origin: "github"`, `relevance_score: 1.0` - `title: "<repo>"` (plain, not "<repo> — Architecture" — the ARCHITECTURE is the canonical view of the repo) - `original_path: "github://<owner>/<repo>@<commit_sha>"` - `source_url: https://github.com/<owner>/<repo>/tree/<commit_sha>` - `github_repo_url: "https://github.com/<owner>/<repo>"` - `github_commit_sha: <commit_sha>` - `github_branch: <branch>` - `github_files`: **union** of every file path referenced across all modules (from the parser output) — used by `/research-distill` for matching, not for indexing individual docs. Empty list in global mode. - `authors: ["<owner>"]` (augment from README if a clear author is declared) - `publication: "GitHub"` - `staged_spec_path`: absolute path to the staged `ARCHITECTURE.md` (builder copies from here) Do NOT emit separate entries for module docs — they live alongside ARCHITECTURE in the repo subfolder and are reached via its links. Once the seed list is assembled (including the GitHub entries), write it to `<research_dir>/seeds.json` (shape: `{"seeds": [ ... ]}`) so the Builder Subagent (Step 6) can consume it without going back through the orchestrator's context. Use the content from seed URIs to enrich your understanding of the topic. Extract additional key themes, terminology, and concepts from them. In discovery modes, these inform the search queries you generate in Step 3; in seed-only modes, they inform the per-source wiki pages and overview/synthesis updates. Summarize your understanding back to the user in 2-3 sentences so they can correct you if needed. ## Step 2 — Configure discovery depth Skip this step entirely for `query`, `append`, and seed-only `init`. Those modes do not run research rounds; set `rounds_completed: 0` for seed-only ingest. This step applies to `deep` mode (and `init` with explicit discovery). Discovery runs at one of exactly three **depth presets** — no free-form round/query counts: | Preset | Rounds | Queries per round | Total queries | |---|---|---|---| | `fast` | 1 | `[3]` | 3 | | `light` | 2 | `[3, 2]` (3 then 2) | 5 | | `deep` | 3 | `[3, 3]` (3 each round) | 9 | **Choosing the preset:** - If the deep-research gate (Step 0) already captured the preset, use it — don't ask again. - Otherwise pre-select from the user's wording: "quick" / "fast" / "shallow" → `fast`; "exhaustive" / "comprehensive" / "thorough" / "deep dive" → `deep`. - If the wording doesn't pin a preset, **ask the user to pick** `fast` / `light` / `deep` (default `light`) before starting rounds. Also capture the **topic slug** — a short kebab-case name for the output directory (suggest one based on the topic). Capture the choice as `total_rounds` and `queries_per_round = (round1_count, subsequent_count)` per the table above (`fast` → `total_rounds=1, (3,)`; `light` → `total_rounds=2, (3, 2)`; `deep` → `total_rounds=3, (3, 3)`), plus `topic_slug`. ## Step 3 — Generate initial search queries Run this step only for `deep` or `init` with explicit discovery. Seed-only modes skip directly to Step 6 after Step 1. Based on the brain dump, generate **exactly `queries_per_round[0]` queries** (3 for every preset) that approach the topic from different angles. The goal is focused breadth - enough to discover missing context without turning a simple request into a long research run. Think about: - **Direct terms**: The obvious keywords - **Related concepts**: Adjacent ideas the user might have notes on - **Synonyms and alternate framings**: Different ways the same idea might be expressed - **Specific entities**: People, tools, frameworks mentioned in the brain dump - **Broader context**: The domain or field this sits within Write these queries down before spawning subagents — you'll refine them in later rounds. ## Step 3b — Discover NotebookLM notebooks Run this step only for `deep` or `init` with explicit discovery. Seed-only modes do not query NotebookLM. The source coverage rule for discovery modes is non-negotiable: **every discovery run queries every NotebookLM notebook, every time. No heuristic filtering.** This is intentional - we cast the widest possible net only when the user chose deep discovery, and dedup consolidates the findings at the end (nothing is filtered out). 1. **Check presence, then authentication.** If `nlm` was MISSING in Step 0.5, skip this entire step: warn "⚠️ `nlm` CLI not found — skipping NotebookLM (see the `nlm-skill`)", set `notebook_ids = []`, and continue. If `nlm` is present, run `nlm login --check`. If that fails, log a loud warning that NotebookLM will be skipped for this run (auth expired/absent), set `notebook_ids = []`, and continue. Do NOT silently skip in either case — the user needs to know NLM was offline so they can decide whether to re-run after installing/fixing it. 2. **List notebooks**: Run `nlm notebook list --json` to get all available notebooks with their IDs, titles, and source counts. 3. **Filter out empty notebooks only** (source_count = 0) — there's nothing to search. Every other notebook is included. 4. **Build `notebook_ids`**: A list of `{id, title}` objects covering every non-empty notebook. Pass it to every researcher subagent. 5. **User-specified notebooks**: If the user referenced specific notebooks in their brain dump (Step 1), they are already in `notebook_ids` (everything is). Note them in the report so the user knows they were prioritized in the search angles. Cost trade-off: querying all notebooks scales with the size of the user's NLM library. For libraries above ~30 notebooks this can be slow — accept the cost; the researchers' `relevance` tags rank the results and dedup consolidates them. If it becomes consistently painful, this is where a `--scope` knob would land (out of scope for now). ## Step 4 — Run research rounds Run this step only for `deep` or `init` with explicit discovery. For each round, spawn **one Research Subagent per query** in parallel using the Agent tool. Each subagent follows the instructions in `agents/researcher.md` (read that file and pass its content as the subagent prompt, along with the specific query and context). The subagent prompt should include: - The search query to execute - The ov …(truncated)