Deep Research

Deep Research

Research the problem thoroughly before writing code. Understand what's known, what's been tried, and what approaches exist.

When to Use

• Starting a new task or problem
• Stuck after multiple evals without improvement
• Pivoting to a fundamentally different approach
• The problem involves domain-specific knowledge you're unfamiliar with

Notes Directory Structure

notes/
├── index.md              ← table of contents for research/ and experiments/
├── raw/                  ← saved web pages, paper excerpts (immutable, never edit)
├── research/             ← your synthesized findings (link back to raw/)
│   └── _coverage.md      ← the research coverage ledger (dimensions × covered/partial/missing)
└── experiments/          ← eval reflections and results (written by reflect heartbeat)

Process

1. Understand the Problem — and Map the Research Space

Read the task description and key files. Identify what's being optimized, what the constraints are, and what makes it hard. Check coral log and {shared_dir}/notes/ for prior work.

Then decompose the problem into 4–8 research dimensions — the distinct things a team would need to understand to win this task. Derive them from the task, don't pull them from a fixed list. Useful starting prompts (not a required set): prior art / SOTA methods, mechanism or theory, implementation / libraries, the evaluation & grader surface, failure modes, adjacent fields. Drop the ones that don't apply; add task-specific ones that do.

Record them in the coverage ledger at {shared_dir}/notes/research/_coverage.md — the team's map of what's been researched and what hasn't. If it doesn't exist, create it with every dimension missing; if it does, read it first and target the gaps rather than re-covering what's done:

# Research Coverage — <task name>
<!-- Owned by the research team. Update on every research pass. Dimensions are
     derived from THIS task, not a fixed list. Status: covered | partial | missing -->

| Dimension (what to understand)  | Status  | Note                                | Last touched |
|---------------------------------|---------|-------------------------------------|--------------|
| Prior art / SOTA methods        | missing | —                                   | —            |
| Failure modes of approach X     | missing | —                                   | —            |
| Evaluation surface / grader     | missing | —                                   | —            |

This ledger is what turns re-research into gap-targeting: every pass below updates one or more rows instead of duplicating work. _coverage.md is a team meta-file (the _ prefix keeps the index/link tooling from treating it as a note) — one per task, updated in place, never forked.

2. Search — Cast a Wide Net, Then Focus

Broad survey — search for the problem class:

• "[problem domain] state of the art methods"
• "[problem domain] survey paper"
• "[problem domain] benchmark comparison"

Specific techniques — once you identify promising approaches:

• "[technique name] vs [alternative] comparison"
• "[technique name] implementation details"
• "[technique name] python library"

Practical implementations — find code and libraries:

• "[problem] python implementation github"
• "[problem] open source solution"

Do 3-5 focused searches. When reading papers and articles, focus on methodology and results tables — how did they solve it, and what performance did they achieve?

Then take one hop out. From your two or three best sources, follow the citation graph one step in each direction: the works they cite (backward — this surfaces the seminal paper the field builds on) and the works that cite them (forward — this surfaces the recent work that extends or contests them). Neither reliably shows up in a keyword sweep. Use WebFetch on a paper's reference list or its Semantic Scholar / Google Scholar "cited by" page, and fold anything new and on-topic into your set before you start writing.

3. Save Raw Sources — Retrieve First, Then Write

The hard rule: a claim you can't point to a saved source for is a claim, not a finding. Retrieve the source before you write the note that leans on it — even when you know the answer cold, fetching the actual page is a few seconds and it's the difference between a citation and a memory of a citation. Never write a number, a benchmark result, or a "X beats Y" into a research note that isn't backed by a file in raw/. The grounding check in step 6 enforces this mechanically.

For every useful source, save the raw content so it can be verified later:

{shared_dir}/notes/raw/source-name.md

Use WebFetch to get the full page, then write it to notes/raw/. These are immutable — never edit raw sources, only reference them from research notes.

When the source is not a plain web article (paper PDF, GitHub repo, video, conference talk, internal docs, chat log…), see references/source-types.md for capture procedure, what to extract, and the right frontmatter fields per type. Generic WebFetch only handles ~half of real research inputs cleanly.

When WebFetch fails, sources contradict, search returns nothing useful, or you find an existing-but-stale note covering your topic, see references/failure-modes.md for diagnosis and recovery procedures.

4. Compare Approaches

Identify 2-4 candidate approaches. For each, document:

• What it is — one-sentence description
• Why it might work — connection to the problem structure
• Known limitations — when it fails or scales poorly
• Estimated complexity — how hard is it to implement?
• Evidence — papers, benchmarks, or reasoning supporting it
• Raw source — link to notes/raw/ entry

Pick your approach based on strength of evidence, implementation feasibility, and iteration potential. Proven methods beat novel ideas for first attempts.

"No such approach exists" is a valid finding. If the honest answer after a real search is that nobody has solved this, or the technique you expected to find was tried and abandoned, write that down — it saves the next agent the same dead-end search. Don't manufacture a weak candidate to fill the table; a documented negative result is more useful than a padded one.

5. Write Research Notes

Summarize your findings in {shared_dir}/notes/research/. For each technique or approach, note:

• What it is and how it works
• Expected trade-offs
• Key parameters and pitfalls
• Links back to raw sources (e.g., see [raw/paper-name.md](../raw/paper-name.md))

Keep notes specific and actionable. "X might work" is weak. "X reduces Y by 30% when Z > 10 (see raw/paper-name.md)" is useful. See references/research-note-template.md for a structured format.

After writing or substantially updating a note, spawn the Synthesis Reviewer subagent to verify grounding. The reviewer reads the note alongside its linked raw sources and returns a per-claim verdict (grounded / partially-grounded / inferred / contradicted / unverifiable), saved next to the note as <slug>.review.json — useful because the author of a synthesis cannot objectively grade its own grounding. See agents/synthesis-reviewer.md for inputs and output schema.

Soft gate: a note is only allowed confidence: high in its frontmatter after it has passed synthesis review — high confidence is earned by an independent grounding check, not self-declared. (The step-6 grounding check flags any confidence: high note that has no .review.json.) For low/medium notes the review is optional but still worth it when the note synthesizes 3+ raw sources, or when a later agent is auditing older notes during organize-files.

6. Update the Index, the Coverage Ledger, and Check Grounding

Create or update {shared_dir}/notes/index.md. The index only lists research notes and experiment notes — not raw sources:

# Notes Index

## Research
- [technique-a](research/technique-a.md) — one-line summary
- [technique-b](research/technique-b.md) — one-line summary

## Experiments
- (none yet)

## Open Questions
- What hasn't been tried?

Raw sources are accessed by following links inside research notes, not through the index.

After writing a new note, run the link resolver so existing notes pick up cross-references to it:

python .coral/public/skills/organize-files/scripts/resolve_links.py {shared_dir}/notes/ --new <new-slug>

The --new flag scans every existing note for plain-text mentions of the new title and wraps them as [[wikilinks]] — without this, manual cross-referencing decays as the notes directory grows.

Update the coverage ledger. For each dimension you touched this pass, flip its row in {shared_dir}/notes/research/_coverage.md to partial or covered, link the note, and stamp the eval count. A pass that found nothing new still updates the row (it's now been looked at) rather than leaving it missing. Update in place — never fork a second ledger.

Run the grounding check before you consider the pass done:

python {shared_dir}/skills/deep-research/scripts/check_grounding.py {shared_dir}/notes

It reads only files already on disk (no network) and flags ungrounded findings: research notes that cite no raw/ source, links to raw/ files that don't exist, raw sources missing their source_url / source_type / captured frontmatter, confidence: high notes that skipped synthesis review, and _coverage.md rows linking to a note that doesn't exist (rename a note and the ledger silently claims coverage it no longer has). Fix what it finds — an ungrounded note pollutes the base for every agent that reads it.

It also reports an advisory uncited-claim count. That one is a deliberately noisy heuristic — treat it as a prompt to re-read those lines, not a gate. It is not folded into grounding_score.

Maintaining Notes Across Sessions

Research notes evolve as new raw sources arrive and old ones decay. A few rules keep the synthesis honest as the corpus grows.

Multi-source synthesis — re-write from ALL contributors

When 2+ raw sources inform the same topic, the research note must draw from every linked source, not just the most recent one. On a follow-up research pass that finds a new source covering an existing topic:

• Update the existing note, don't fork. research/topic-v2.md is wrong — there should be one note per topic.
• Re-read each linked raw source and rewrite the synthesis from the full set, not just the new one.
• Append the new source to the ## References section.

If you only re-synthesize from the new source, you silently drop evidence from the old ones — which means re-research can quietly reduce the note's grounding instead of strengthening it.

Stale or invalid sources — freeze, don't overwrite

When a raw source becomes invalid (link rot, retraction, supersession by a newer paper):

• Don't rewrite the note immediately. Add needs-reverification: [list of claims] to the frontmatter and move on.
• Only rewrite once you can confirm which claims survive on the remaining sources.
• If the note loses all its supporting sources, set superseded: true rather than deleting the file — it preserves the audit trail for future agents who might rediscover the topic.

Partial re-verification preserves combined work

If a note synthesizes A + B + C and you only have time to re-verify against B, leave the note body alone. A partial rewrite that keeps only B's perspective drops the synthesis from A and C. Either re-verify against the full source set or note the partial check in the frontmatter (partially-verified: [B]) without touching the body.

Principles

• Retrieve first, then write — a claim you can't point to a saved source for is a claim, not a finding
• Save raw sources — summaries can be wrong, raw sources are ground truth
• Target the gaps — read _coverage.md first; research what's missing, don't re-cover what's covered
• Breadth before depth — survey 3+ approaches before committing to one
• Compare before committing — always evaluate 2-4 candidates, don't latch onto the first result
• Build on what exists — check notes and past attempts first
• Cite your sources — link research notes back to notes/raw/, and run the grounding check before you're done
• Don't over-research — 3-5 searches, write notes, start coding