Deep Research

Deep Research Skill

Perform systematic, multi-round deep research with real web search and content extraction tools, adaptive planning, confidence tracking, and structured report generation.

When to use this skill

Use this skill when the user asks to:

• Research a topic in depth or investigate something
• Create a comprehensive, evidence-based research report
• Synthesize multiple perspectives on a complex question
• Analyze a topic across factual, comparative, economic, temporal, or other angles
• Produce a structured report with proper citations and source triangulation

Do not use this skill for simple factual lookups (use a single web search instead), real-time monitoring, or generating primary data.

Quick start

For a full automated research run from this skill directory:

python scripts/research.py "your research topic" --breadth 5 --depth 3

For machine-readable output:

python scripts/research.py "your research topic" --json --output results.json

Individual tools:

python scripts/search.py "search query" --engine auto --num 10 --json
python scripts/fetch_content.py https://example.com --max-chars 5000 --json

Core workflow

The skill follows a Plan → Search → Fetch → Reflect → Report loop, using the bundled scripts as tools at each stage.

1. Initialize — parse request and confirm configuration

Extract a configuration from the user's request:

• Topic — the core research question.
• Breadth — parallel queries per round (default 5; range 3–15).
• Depth — maximum research iterations (default 3; range 1–7).
• Report Format — comprehensive | executive | technical | market | academic.
• Language — output language (default: same as user query).
• User Constraints — scope limits, time range, domain focus, specific questions.

Confirm the configuration with the user before proceeding.

2. Plan — decompose topic into sub-questions

Decompose the topic into 5–10 research sub-questions using the 10-Angle Framework:

Produce a Research Plan table mapping each sub-question to query strategy, target source types, and initial confidence (0.0).

3. Iterative research loop (repeat up to depth rounds)

Each round uses the bundled tools to search, fetch, and reflect.

3a. REASON — select queries

Examine sub-question confidence scores. Allocate more queries to lower-confidence areas. Reserve 1–2 queries for lateral discovery. Diversify source types.

Generate breadth search queries for this round. Write them to a JSON list.

3b. ACT — execute searches

Run the search tool:

python scripts/search.py --queries queries.json --engine auto --num 8 --json > search_results.json

Or for a single query:

python scripts/search.py "electric vehicle lifecycle emissions" --num 10 --json

The search tool supports multiple backends (auto-detected by available API keys):

• DuckDuckGo (default, no API key needed)
• Google Custom Search (set GOOGLE_API_KEY + GOOGLE_CX)
• Bing Web Search (set BING_API_KEY)
• SerpAPI (set SERPAPI_KEY)

Each result includes: URL, title, snippet, source type (academic/government/industry/news/blog/web), and credibility rating (high/medium/low) based on domain heuristics.

3c. ACT — fetch top source content

For the most promising results, fetch full page content:

python scripts/fetch_content.py https://example.com/article --max-chars 5000 --json

Or batch fetch:

python scripts/fetch_content.py --urls top_urls.json --max-chars 5000 --json > content.json

The fetch tool extracts readable text from HTML, stripping navigation, ads, and boilerplate. It returns: URL, title, meta description, extracted text content, and fetch status.

3d. OBSERVE — synthesize round findings

After collecting search results and fetched content:

1. Cluster findings by sub-question / theme.
2. Triangulate — mark claims supported by ≥ 2 independent sources as corroborated; single-source claims as unverified.
3. Flag conflicts — where sources disagree, record both sides with evidence quality notes.
4. Merge & compress — deduplicate; keep the most authoritative version of each fact.
5. Extract key data — specific numbers, dates, names, quotes.

Output a per-round Findings Summary (compressed, ≤ 500 words).

3e. REFLECT — confidence scoring and gap analysis

Update confidence for each sub-question:

Perform gap analysis:

1. Which sub-questions are still below 0.7?
2. Are there new sub-questions raised by this round's findings?
3. Are there conflicts requiring targeted verification?
4. Were any assumptions in the plan invalidated?

Update the Research Plan accordingly.

3f. REVISE — decide whether to continue

Continue if: round < depth AND at least one sub-Q < 0.7 AND meaningful new info found AND viable queries remain.

Early-stop if: all sub-Qs ≥ 0.7 OR last round added < 10% new info (saturation) OR remaining gaps are unanswerable via web search.

Adapt breadth: narrow if many irrelevant results; widen if new angles emerged.

4. Synthesize — organize and resolve

• Cluster findings into 3–7 major themes.
• Order evidence by strength: corroborated → supported → single-source → contested → unverified.
• Resolve conflicts by comparing credibility, methodology, context, and looking for meta-analyses.
• Produce a confidence summary table.

5. Report — generate structured output

Generate a structured report:

1. Executive Summary (200–300 words) — scope, methodology, key findings, confidence, conclusions.
2. Introduction (150–250 words) — background, objectives, scope.
3. Main Analysis — 3–7 thematic sections with inline citations [N], tables, conflicts with evidence weights.
4. Patterns and Implications (200–400 words) — cross-cutting themes, practical implications.
5. Conclusion (150–250 words) — answers, confidence assessment, next steps.
6. Sources — numbered bibliography with author, title, publication, date, URL.

Append a Confidence Note: overall score, low-confidence areas, and caveats.

Running the full pipeline as a single command

For standalone use or automated runs:

python scripts/research.py "impact of remote work on housing markets" \
    --breadth 5 --depth 3 --engine auto \
    --fetch-top 3 --max-chars 3000 \
    --output report.md

With state persistence (allows resuming interrupted research):

python scripts/research.py "topic" --depth 3 --state-file state.json --output report.md

The orchestrator script handles the full loop automatically when run standalone. When used through an LLM agent, the agent should call the individual tools (search.py, fetch_content.py) directly at each step to maintain control over query generation and synthesis.

LLM Agent Execution Protocol

When you are acting as an AI agent applying this skill:

1. Autonomy: DO NOT ask the user to run the scripts. You must use the run_in_terminal tool to execute python scripts/search.py and python scripts/fetch_content.py yourself.
2. Context Management: Direct JSON outputs into temporary files if they are too large (e.g., > temp_results.json), then read them back to synthesize.
3. Pacing: Announce your plan to the user, execute the loop iteratively, synthesize the results, and only notify completion when the final report is ready.
4. Resilience: If a search engine fails (e.g., rate limits or API key missing), automatically fallback to --engine duckduckgo, which requires no configuration. Ensure you check for Python missing dependency errors and prompt the user to install them (e.g., pip install -r requirements.txt) if they occur.

Tool reference

scripts/search.py — Web search

Environment variables for API-based engines:

• GOOGLE_API_KEY + GOOGLE_CX — Google Custom Search
• BING_API_KEY — Bing Web Search API
• SERPAPI_KEY — SerpAPI

Output fields per result: url, title, snippet, source_type, credibility, fingerprint, fetched_at.

scripts/fetch_content.py — Content extraction

Output fields: ok, url, final_url, status_code, title, description, content, content_length, fetched_at, error.

scripts/research.py — Full research orchestrator

scripts/common.py — Shared utilities

Provides: URL normalization, domain extraction, text cleaning, HTML stripping, credibility heuristics, source type detection, deduplication, Markdown formatting helpers, and JSON I/O.

Evidence and confidence requirements

Each reported claim should trace back to a source card with:

• url — source URL
• title — page title
• credibility — high, medium, or low (auto-scored by domain heuristics from common.py, refined by LLM assessment)
• source_type — academic, government, industry, news, blog, or web
• snippet or fetched_content — the supporting evidence

Confidence rules:

• High — multiple independent high-credibility sources corroborate.
• Medium — one strong signal or multiple weaker signals.
• Low — a single weak or generic source.

Never report a claim with no source.

Citation rules

• Use [N] inline format. Every factual claim MUST have a citation.
• Use the most authoritative source. Multiple citations for corroborated claims: [1][3].
• Direct quotes in quotation marks.
• No single source > 25% of citations. Sources must span ≥ 3 domain types.

Edge cases

• Broad topic → ask user to narrow, or propose 3–5 angles.
• Fast-moving topic → prioritize recency (after:2024 in queries), flag short shelf-life.
• Niche topic → expand to preprints, technical blogs; note limited sourcing.
• Controversial topic → present all perspectives, extra triangulation, flag sensitivity.
• Insufficient data → report honestly, lower confidence, suggest alternative methods.
• Search engine rate limits → increase --delay, switch engine, or reduce --num.

Output format

Default report length by depth:

References

• references/source-credibility.md — ACAP credibility framework and scoring.
• references/strategy-guidance.md — query refinement, gap analysis, compression.
• references/report-template.md — structured report templates by format.
• scripts/ — executable tools and shared utilities.