Search Tips
Accumulated guidance for web research using Exa, Firecrawl CLI, and Reddit MCP tools. These
are starting points, not rigid rules -- think strategically about each situation and
adapt. If a different approach makes more sense for what you're trying to do, go with it.
Run npx firecrawl-cli <command> --help to check available options beyond what's documented
here. Reference files cover tool-specific deep dives -- see bottom of this file.
Setup
Before starting research, load the required MCP tools using ToolSearch:
- Exa tools --
web_search_advanced_exa and get_code_context_exa
- Reddit tools --
get_top_posts, get_post_comments, get_reddit_post, get_subreddit_info
Firecrawl CLI (npx firecrawl-cli) runs via Bash -- no MCP setup needed. Load only what
the task requires.
The Research Cycle
Prefer Exa and Firecrawl over built-in WebSearch/WebFetch.
Research alternates between searching (discovering sources) and fetching (extracting
content from them). Find promising leads, read the best ones, refine your understanding,
search again.
Searching
Finding sources you don't have yet.
- Exa search (
web_search_advanced_exa) -- primary tool for web discovery. Natural
language queries, add filters as needed (domains, dates, categories).
- Exa code context (
get_code_context_exa) -- programming topics. Worth trying before
general Exa search for technical/code tasks -- surfaces repos, packages, and docs.
- Firecrawl CLI search (
npx firecrawl-cli search) -- Google-powered keyword search.
Useful when keyword matching works better than Exa's semantic approach, for site-scoped
queries (site:reddit.com {query}), and for content-type filtering (--categories research
for academic, --sources news, --tbs qdr:w for time).
Default Exa search pattern: Default to enableHighlights: true and textMaxCharacters: 1.
This returns quoted passages from actual page text while preventing the MCP server from
flooding context with full text. Use highlightsPerUrl and highlightsNumSentences to
control volume if needed.
Fetching
Extracting content from a source you've identified.
- Firecrawl CLI scrape (
npx firecrawl-cli scrape "<url>" --only-main-content) -- primary
tool for reading a known URL. The flag strips nav/sidebars to save tokens.
- Reddit MCP (
get_post_comments, get_reddit_post) -- for reading Reddit threads.
Firecrawl can't scrape reddit.com directly.
- Firecrawl CLI map (
npx firecrawl-cli map "<url>" --search "query") -- discover URLs
on a site (useful when you need to find the right page, or when scrape returns empty).
Adapting the Workflow
The defaults above won't always be right. Some common deviations:
- Exa full text as a scraping fallback -- some sites are blocked or inaccessible via
Firecrawl (LinkedIn, Twitter/X, etc.), but Exa often has the full page text in its index.
Drop both
enableHighlights and textMaxCharacters: 1 to get the complete text. Be aware
this can produce large responses.
--only-main-content can strip too much -- if you got empty or partial results, retry
without the flag. Known to fail on Future plc sites, Blogspot, and GDPR-heavy sites. See
Content Extraction below.
The reference files cover more edge cases -- scraping issues, category restrictions, and
academic search.
Search Strategy
How Exa Works
Exa is a neural/semantic search engine. It uses embeddings to understand meaning.
- Natural questions or statements work best -- Exa finds pages that answer them
- Longer, more specific queries work BETTER -- unlike keyword-based search
- Keyword lists tend to confuse the semantic model
Good: "What do professional reviewers say are the most reliable dishwasher brands in 2025?"
Bad: "best dishwasher 2025 reliable"
Query Reformulation
For broad topics, Exa's additionalQueries parameter can automate this -- it bundles query
variations in a single call at no extra cost (see references/exa-tips.md). For manual
reformulation, try generating 3-5 query variations:
| Technique |
What It Does |
Example |
| Paraphrase |
Same meaning, different words |
"RAG failures" -> "problems in RAG systems" |
| Decompose |
Break into sub-questions |
"Why fail?" -> "Why return irrelevant docs?" |
| Scope shift |
Broader context or narrower specifics |
"Challenges in production AI search" |
| Perspective shift |
Different viewpoints |
User vs expert vs critic view |
| Temporal framing |
Target different time periods |
"Recent 2024-2025" vs "foundational" |
Domain Filtering
Try includeDomains when the authoritative site for a topic is known -- faster and less noisy
than broad search. Try excludeDomains to suppress sites that keep appearing but aren't
useful (e.g., exclude youtube.com when video pages crowd out needed editorial content about
YouTube creators/content).
Searching by Content Type
Match your research target to the right approach. Reference files have full strategies.
- Social sentiment / opinions -- Exa
tweet for Twitter; site:reddit.com via Firecrawl
search + Reddit MCP for discussions. See references/twitter.md, references/reddit.md.
- People / companies -- Exa
people or company category for discovery, then broaden.
See references/people-companies.md.
- Academic papers -- Exa
research paper category; academic APIs for structured data.
See references/academic-search.md.
- Code / GitHub --
get_code_context_exa for code; gh api for repo data.
See references/code-github.md.
- News -- Exa
news category with date filters; Firecrawl CLI
search --sources news --tbs qdr:w for Google News with time filtering.
- Financial reports -- Exa
financial report with domain/date filtering.
- Personal blogs / independent takes -- Exa
personal site for practitioner opinions,
blog posts, and independent analysis. Full parameter support (unlike most specialized
categories). See references/personal-sites.md.
Some categories reject certain parameters (400/500 errors). See references/exa-tips.md.
Content Extraction
Always use --only-main-content by default -- it strips nav, sidebars, and footers, saving
significant tokens. If you get empty or partial results, retry without the flag; it's known
to strip article bodies on Future plc sites (iMore, Pocket-lint), GDPR-heavy sites
(StorageReview), and Blogspot blogs -- see references/scraping-issues.md.
For even narrower extraction, use --include-tags (e.g., --include-tags "article",
--include-tags ".post-content").
Scraping Issues
When Firecrawl scrape fails (policy blocks, paywalls, SPA rendering), check
references/scraping-issues.md for per-site workarounds. General fallback: try Exa full text
(drop enableHighlights and textMaxCharacters). For interactive pages that need clicks or
form fills, escalate to npx firecrawl-cli browser (see references/firecrawl-tips.md).
Operational Notes
Parallel call failures: If any tool call in a parallel batch fails, all sibling calls
fail too. Retry individually. Keep failure-prone calls (Reddit MCP) in their own batch.
Rate limits: Retry once after a short pause. If still blocked, try a different tool for the
same intent before giving up -- Exa rate-limited? Try Firecrawl search. Reddit MCP throttled?
Try Exa with includeDomains: ["reddit.com"]. Only note the gap and move on after both the
original tool and an alternative have failed.
Reference Files
Content-type strategies
references/twitter.md -- Exa tweet category, restrictions, query tips, livecrawlreferences/reddit.md -- Discovery via Firecrawl search, Reddit MCP for reading,
batching gotchas, rate limitsreferences/people-companies.md -- Exa people/company categories, LinkedIn, multi-
category approachreferences/code-github.md -- Code search, GitHub repos/issues, gh api, raw URLsreferences/personal-sites.md -- Independent blogs, practitioner opinions, full
parameter support, portfolio explorationreferences/academic-search.md -- Academic domains, free APIs, Firecrawl CLI
Tool reference
references/exa-tips.md -- Category restrictions, additionalQueries,
highlights/summariesreferences/firecrawl-tips.md -- CLI commands (search, scrape, map, crawl, download,
browser), PDF scraping, arxiv extraction, MCP fallback notes
Troubleshooting
references/scraping-issues.md -- main-content detection fixes, paywalls, Discourse
JSON, policy-blocked sites, API workarounds
1---2name: search-tips3description: This skill should be used when performing web research beyond a simple single search -- looking into topics, comparing options, investigating questions, finding recommendations, or any task where effective use of Exa, Firecrawl, and Reddit tools matters. Triggers on "research", "look into", "investigate", "compare", "find out about", "search for", "find information", "what do people think about", "what are the best", "look up", or multi-source search tasks. Also invocable explicitly by deep-research team members via the Skill tool.4---5
6# Search Tips
7
8Accumulated guidance for web research using Exa, Firecrawl CLI, and Reddit MCP tools. These
9are **starting points, not rigid rules** -- think strategically about each situation and
10adapt. If a different approach makes more sense for what you're trying to do, go with it.
11Run `npx firecrawl-cli <command> --help` to check available options beyond what's documented
12here. Reference files cover tool-specific deep dives -- see bottom of this file.
13
14## Setup
15
16Before starting research, load the required MCP tools using ToolSearch:
17
181. **Exa tools** -- `web_search_advanced_exa` and `get_code_context_exa`
192. **Reddit tools** -- `get_top_posts`, `get_post_comments`, `get_reddit_post`, `get_subreddit_info`
20
21Firecrawl CLI (`npx firecrawl-cli`) runs via Bash -- no MCP setup needed. Load only what
22the task requires.
23
24## The Research Cycle
25
26Prefer Exa and Firecrawl over built-in WebSearch/WebFetch.
27
28Research alternates between **searching** (discovering sources) and **fetching** (extracting
29content from them). Find promising leads, read the best ones, refine your understanding,
30search again.
31
32### Searching
33
34Finding sources you don't have yet.
35
36- **Exa search** (`web_search_advanced_exa`) -- primary tool for web discovery. Natural
37 language queries, add filters as needed (domains, dates, categories).
38- **Exa code context** (`get_code_context_exa`) -- programming topics. Worth trying before
39 general Exa search for technical/code tasks -- surfaces repos, packages, and docs.
40- **Firecrawl CLI search** (`npx firecrawl-cli search`) -- Google-powered keyword search.
41 Useful when keyword matching works better than Exa's semantic approach, for site-scoped
42 queries (`site:reddit.com {query}`), and for content-type filtering (`--categories research`
43 for academic, `--sources news`, `--tbs qdr:w` for time).
44
45**Default Exa search pattern:** Default to `enableHighlights: true` and `textMaxCharacters: 1`.
46This returns quoted passages from actual page text while preventing the MCP server from
47flooding context with full text. Use `highlightsPerUrl` and `highlightsNumSentences` to
48control volume if needed.
49
50### Fetching
51
52Extracting content from a source you've identified.
53
54- **Firecrawl CLI scrape** (`npx firecrawl-cli scrape "<url>" --only-main-content`) -- primary
55 tool for reading a known URL. The flag strips nav/sidebars to save tokens.
56- **Reddit MCP** (`get_post_comments`, `get_reddit_post`) -- for reading Reddit threads.
57 Firecrawl can't scrape reddit.com directly.
58- **Firecrawl CLI map** (`npx firecrawl-cli map "<url>" --search "query"`) -- discover URLs
59 on a site (useful when you need to find the right page, or when scrape returns empty).
60
61### Adapting the Workflow
62
63The defaults above won't always be right. Some common deviations:
64
65- **Exa full text as a scraping fallback** -- some sites are blocked or inaccessible via
66 Firecrawl (LinkedIn, Twitter/X, etc.), but Exa often has the full page text in its index.
67 Drop both `enableHighlights` and `textMaxCharacters: 1` to get the complete text. Be aware
68 this can produce large responses.
69- **`--only-main-content` can strip too much** -- if you got empty or partial results, retry
70 without the flag. Known to fail on Future plc sites, Blogspot, and GDPR-heavy sites. See
71 Content Extraction below.
72
73The reference files cover more edge cases -- scraping issues, category restrictions, and
74academic search.
75
76## Search Strategy
77
78### How Exa Works
79
80Exa is a **neural/semantic search engine**. It uses embeddings to understand meaning.
81
82- **Natural questions or statements work best** -- Exa finds pages that answer them
83- **Longer, more specific queries work BETTER** -- unlike keyword-based search
84- **Keyword lists tend to confuse** the semantic model
85
86Good: "What do professional reviewers say are the most reliable dishwasher brands in 2025?"
87Bad: "best dishwasher 2025 reliable"
88
89### Query Reformulation
90
91For broad topics, Exa's `additionalQueries` parameter can automate this -- it bundles query
92variations in a single call at no extra cost (see `references/exa-tips.md`). For manual
93reformulation, try generating 3-5 query variations:
94
95| Technique | What It Does | Example |
96| --------------------- | ------------------------------------- | -------------------------------------------- |
97| **Paraphrase** | Same meaning, different words | "RAG failures" -> "problems in RAG systems" |
98| **Decompose** | Break into sub-questions | "Why fail?" -> "Why return irrelevant docs?" |
99| **Scope shift** | Broader context or narrower specifics | "Challenges in production AI search" |
100| **Perspective shift** | Different viewpoints | User vs expert vs critic view |
101| **Temporal framing** | Target different time periods | "Recent 2024-2025" vs "foundational" |
102
103### Domain Filtering
104
105Try `includeDomains` when the authoritative site for a topic is known -- faster and less noisy
106than broad search. Try `excludeDomains` to suppress sites that keep appearing but aren't
107useful (e.g., exclude `youtube.com` when video pages crowd out needed editorial content about
108YouTube creators/content).
109
110### Searching by Content Type
111
112Match your research target to the right approach. Reference files have full strategies.
113
114- **Social sentiment / opinions** -- Exa `tweet` for Twitter; `site:reddit.com` via Firecrawl
115 search + Reddit MCP for discussions. See `references/twitter.md`, `references/reddit.md`.
116- **People / companies** -- Exa `people` or `company` category for discovery, then broaden.
117 See `references/people-companies.md`.
118- **Academic papers** -- Exa `research paper` category; academic APIs for structured data.
119 See `references/academic-search.md`.
120- **Code / GitHub** -- `get_code_context_exa` for code; `gh api` for repo data.
121 See `references/code-github.md`.
122- **News** -- Exa `news` category with date filters; Firecrawl CLI
123 `search --sources news --tbs qdr:w` for Google News with time filtering.
124- **Financial reports** -- Exa `financial report` with domain/date filtering.
125- **Personal blogs / independent takes** -- Exa `personal site` for practitioner opinions,
126 blog posts, and independent analysis. Full parameter support (unlike most specialized
127 categories). See `references/personal-sites.md`.
128
129Some categories reject certain parameters (400/500 errors). See `references/exa-tips.md`.
130
131## Content Extraction
132
133Always use `--only-main-content` by default -- it strips nav, sidebars, and footers, saving
134significant tokens. If you get empty or partial results, retry without the flag; it's known
135to strip article bodies on Future plc sites (iMore, Pocket-lint), GDPR-heavy sites
136(StorageReview), and Blogspot blogs -- see `references/scraping-issues.md`.
137
138For even narrower extraction, use `--include-tags` (e.g., `--include-tags "article"`,
139`--include-tags ".post-content"`).
140
141## Scraping Issues
142
143When Firecrawl scrape fails (policy blocks, paywalls, SPA rendering), check
144`references/scraping-issues.md` for per-site workarounds. General fallback: try Exa full text
145(drop `enableHighlights` and `textMaxCharacters`). For interactive pages that need clicks or
146form fills, escalate to `npx firecrawl-cli browser` (see `references/firecrawl-tips.md`).
147
148## Operational Notes
149
150**Parallel call failures:** If any tool call in a parallel batch fails, all sibling calls
151fail too. Retry individually. Keep failure-prone calls (Reddit MCP) in their own batch.
152
153**Rate limits:** Retry once after a short pause. If still blocked, try a different tool for the
154same intent before giving up -- Exa rate-limited? Try Firecrawl search. Reddit MCP throttled?
155Try Exa with `includeDomains: ["reddit.com"]`. Only note the gap and move on after both the
156original tool and an alternative have failed.
157
158## Reference Files
159
160### Content-type strategies
161- **`references/twitter.md`** -- Exa tweet category, restrictions, query tips, livecrawl
162- **`references/reddit.md`** -- Discovery via Firecrawl search, Reddit MCP for reading,
163 batching gotchas, rate limits
164- **`references/people-companies.md`** -- Exa people/company categories, LinkedIn, multi-
165 category approach
166- **`references/code-github.md`** -- Code search, GitHub repos/issues, gh api, raw URLs
167- **`references/personal-sites.md`** -- Independent blogs, practitioner opinions, full
168 parameter support, portfolio exploration
169- **`references/academic-search.md`** -- Academic domains, free APIs, Firecrawl CLI
170
171### Tool reference
172- **`references/exa-tips.md`** -- Category restrictions, additionalQueries,
173 highlights/summaries
174- **`references/firecrawl-tips.md`** -- CLI commands (search, scrape, map, crawl, download,
175 browser), PDF scraping, arxiv extraction, MCP fallback notes
176
177### Troubleshooting
178- **`references/scraping-issues.md`** -- main-content detection fixes, paywalls, Discourse
179 JSON, policy-blocked sites, API workarounds
majiayu000