Public Web Research at Scale
Pairs with thinking/evidence-grading (which grades the claims) — this skill is about getting
the claims: what to search, what to fetch, in what order, and when to stop.
The failure mode it kills: an agent runs one search, reads the top three SEO results, and produces
a fluent report whose every number traces back to the same content-marketing blog post.
Principle: search to find the primary source, then go read it
Search results are a map to primary sources, not evidence themselves. A listicle saying "the
market is $4.2B" is grade D. Its value is that it may name the analyst firm — and that report,
or the filing, is what you actually cite. Always take one more hop toward the origin.
The chain almost always looks like:
SEO blog post (D) → names a report (C) → report's own page w/ methodology (B) → the filing / dataset / API (A)
Most research stops at hop 1. The entire difference in output quality is hops 2–4.
Step 1 — decompose before searching
Write the question, then break it into sub-questions that each have a factual answer and a
plausible source type. Searching the headline question directly is what lands you in SEO content,
because the headline question is exactly what content marketers target.
Example — "Is there a business in dispatch software for UK trade firms?"
| Sub-question |
Source type |
Expected grade |
| How many UK firms in SIC 43.22, by employee band? |
ONS / Companies House bulk data |
A |
| What do incumbents charge? |
Vendors' own pricing pages |
A |
| What do users complain about? |
Public reviews, forums, trade subreddits |
C, but aggregate = signal |
| Are firms hiring for scheduling roles? |
Job boards |
A (primary observation) |
| Is anyone funded in this space recently? |
Filings, press releases |
A/B |
Write the sub-question table before the first search. It prevents the drift where you research
whatever the internet happens to serve you.
Step 2 — query strategy
- Vary the register. How a practitioner phrases it ≠ how a marketer phrases it. Search both:
dispatch software pricing (marketer) vs how do you schedule 8 vans (practitioner). The second
finds forums, which is where the truth about problems lives.
- Use site/filetype operators to jump straight to primary sources:
site:gov.uk,
site:ons.gov.uk, filetype:pdf annual report, site:sec.gov.
- Search for the negative.
<thing> problems, <thing> alternatives, <thing> cancel,
<vendor> vs. Critical sources are more informative than promotional ones and rank lower.
- Search the specific number you were given, in quotes, to find its origin and see how many
"independent" sources are laundering one press release.
- Date-bound it. Add the year, or restrict recency. Undated pages are grade D on principle.
- Try the practitioner venues directly: trade forums, subreddits, Stack Exchange, association
publications, local trade press. Low SEO competition, high truth density.
Step 3 — fetch discipline
web_search gives titles and snippets; snippets are not evidence. Fetch the page.
- Fetch the actual page before citing it. A snippet is often the opposite of the article's
conclusion.
- Record fetch date and publication date separately. Both matter; a 2019 stat is C at best in 2026.
- If a fetch fails, that is data, not a gap to paper over. Record it as grade E with the reason
(paywall, 403, JS-only, dead link). "I could not verify this" is a legitimate and useful finding.
- Know the tool's limits and say so:
web_fetch returns static HTML only. JS-rendered pages,
infinite-scroll apps, and login-walled content return nothing useful — that's a tooling gap to
report, never "no information exists." Confusing the two is how an agent invents a conclusion.
- Never fabricate a URL or a citation. If you can't produce the link, you don't have the source.
- Respect robots.txt and rate-limit politely (≤1 req/sec/host); cache so you never fetch twice.
Step 4 — triangulation and contradictions
Require two independent sources for any load-bearing claim — and check independence by tracing
each to its origin. Two outlets citing the same press release is one source.
When sources contradict, do not average them and do not silently pick one. Resolve in this order:
- Is one closer to primary? Prefer it.
- Do they measure different things? (Revenue vs bookings; "users" vs "paying users"; different
geography or year.) This is the answer ~70% of the time — the contradiction is definitional.
- Is one stale? Prefer recent, note the trend.
- Does one have a disclosed methodology and an N? Prefer it.
- Unresolved → report the range and the disagreement explicitly. A documented range is an
honest finding; a false point estimate is not.
Triangulate across source types, not just documents. Official data + vendor pricing + user
complaints + hiring activity, all agreeing, is strong. Four blog posts agreeing is one blog post.
Step 5 — stopping rules
Stop when any is true:
- Every sub-question has ≥1 A/B source, and load-bearing ones have two independent.
- New searches return sources you've already seen (saturation — the real signal).
- Remaining unknowns wouldn't change the decision (per
thinking/decision-quality).
- The unknown needs a capability you don't have (browser, paid data, a customer conversation) —
stop and name the capability; that's the finding.
Anti-pattern: researching until it feels thorough. Depth ≠ time spent. The sub-question table
tells you when you're done; feelings don't.
Output format
QUESTION
<the decision this serves>
ANSWER
<2–4 sentences. Confidence stated. Lead with the answer, not the process.>
SUB-QUESTIONS
| Q | Answer | Grade | Source (url, publisher, pub date, fetched) |
CONTRADICTIONS
<what disagreed, how resolved, or the unresolved range>
WHAT I COULD NOT VERIFY
<grade D/E items, failed fetches, tooling gaps — explicitly listed>
WHAT WOULD CHANGE THE ANSWER
<the specific fetch/experiment that would move it>
The last two sections are mandatory. A report without them reads as more certain than it is,
which is the harm this skill exists to prevent.
Failure modes
| Symptom |
Cause |
Fix |
| All sources are blogs |
Searched the headline question |
Decompose; use site: operators for primary data |
| Confident market size, no N |
Number laundered through aggregators |
Search the number in quotes; find origin or strike it |
| "No information available" |
JS-only pages returned empty |
Report tooling gap, don't conclude absence |
| Report is 3,000 words, decision unchanged |
No sub-question table; researched for its own sake |
Apply stopping rules |
| Sources all agree suspiciously |
One origin, many republishers |
Trace independence |
| Only positive sources |
Never searched the negative |
Add problems/alternatives/vs queries |
Boundaries
- Public pages, official APIs, licensed data, public registries only. No login-walled extraction,
no CAPTCHA/paywall bypass (
sales/outbound-compliance governs).
- No fabricated citations, ever. No inferred URLs.
- Findings exit via
surface_finding with a provenance block — nothing graded D/E appears in
a summary field (thinking/evidence-grading, hard rule).
1---2name: public-web-research-at-scale3description: "Run a multi-source web investigation that produces an auditable answer instead of a confident guess — query strategy, source hierarchy, fetch discipline, contradiction handling, and knowing when to stop. Use for any question needing more than one lookup: market sizing, company investigation, technical due diligence, \"is this real\". Triggers on \"research\", \"find out\", \"how big is\", \"who are the\", \"is it true that\", \"due diligence\", \"market size\", \"investigate\", \"look into\"."4metadata:5 source: https://skillmd.com/skills/cspoerer62/public-web-research-at-scale6---78# Public Web Research at Scale910Pairs with `thinking/evidence-grading` (which grades the claims) — this skill is about **getting**11the claims: what to search, what to fetch, in what order, and when to stop.1213The failure mode it kills: an agent runs one search, reads the top three SEO results, and produces14a fluent report whose every number traces back to the same content-marketing blog post.1516## Principle: search to find the primary source, then go read it1718Search results are a **map to primary sources, not evidence themselves.** A listicle saying "the19market is $4.2B" is grade D. Its value is that it may name the analyst firm — and *that* report,20or the filing, is what you actually cite. **Always take one more hop toward the origin.**2122The chain almost always looks like:2324```25SEO blog post (D) → names a report (C) → report's own page w/ methodology (B) → the filing / dataset / API (A)26```2728Most research stops at hop 1. The entire difference in output quality is hops 2–4.2930## Step 1 — decompose before searching3132Write the question, then break it into **sub-questions that each have a factual answer and a33plausible source type.** Searching the headline question directly is what lands you in SEO content,34because the headline question is exactly what content marketers target.3536Example — "Is there a business in dispatch software for UK trade firms?"3738| Sub-question | Source type | Expected grade |39|---|---|---|40| How many UK firms in SIC 43.22, by employee band? | ONS / Companies House bulk data | A |41| What do incumbents charge? | Vendors' own pricing pages | A |42| What do users complain about? | Public reviews, forums, trade subreddits | C, but aggregate = signal |43| Are firms hiring for scheduling roles? | Job boards | A (primary observation) |44| Is anyone funded in this space recently? | Filings, press releases | A/B |4546**Write the sub-question table before the first search.** It prevents the drift where you research47whatever the internet happens to serve you.4849## Step 2 — query strategy5051- **Vary the register.** How a practitioner phrases it ≠ how a marketer phrases it. Search both:52 `dispatch software pricing` (marketer) vs `how do you schedule 8 vans` (practitioner). The second53 finds forums, which is where the truth about problems lives.54- **Use site/filetype operators** to jump straight to primary sources: `site:gov.uk`,55 `site:ons.gov.uk`, `filetype:pdf annual report`, `site:sec.gov`.56- **Search for the negative.** `<thing> problems`, `<thing> alternatives`, `<thing> cancel`,57 `<vendor> vs`. Critical sources are more informative than promotional ones and rank lower.58- **Search the specific number** you were given, in quotes, to find its origin and see how many59 "independent" sources are laundering one press release.60- **Date-bound it.** Add the year, or restrict recency. Undated pages are grade D on principle.61- **Try the practitioner venues directly**: trade forums, subreddits, Stack Exchange, association62 publications, local trade press. Low SEO competition, high truth density.6364## Step 3 — fetch discipline6566`web_search` gives titles and snippets; **snippets are not evidence.** Fetch the page.6768- **Fetch the actual page** before citing it. A snippet is often the opposite of the article's69 conclusion.70- **Record fetch date and publication date separately.** Both matter; a 2019 stat is C at best in 2026.71- **If a fetch fails, that is data, not a gap to paper over.** Record it as grade E with the reason72 (paywall, 403, JS-only, dead link). "I could not verify this" is a legitimate and useful finding.73- **Know the tool's limits and say so**: `web_fetch` returns static HTML only. JS-rendered pages,74 infinite-scroll apps, and login-walled content return nothing useful — that's a **tooling gap to75 report**, never "no information exists." Confusing the two is how an agent invents a conclusion.76- **Never fabricate a URL or a citation.** If you can't produce the link, you don't have the source.77- Respect robots.txt and rate-limit politely (≤1 req/sec/host); cache so you never fetch twice.7879## Step 4 — triangulation and contradictions8081Require **two independent sources** for any load-bearing claim — and check independence by tracing82each to its origin. Two outlets citing the same press release is **one** source.8384When sources contradict, **do not average them and do not silently pick one.** Resolve in this order:851. Is one closer to primary? Prefer it.862. Do they measure different things? (Revenue vs bookings; "users" vs "paying users"; different87 geography or year.) This is the answer ~70% of the time — the contradiction is definitional.883. Is one stale? Prefer recent, note the trend.894. Does one have a disclosed methodology and an N? Prefer it.905. **Unresolved → report the range and the disagreement explicitly.** A documented range is an91 honest finding; a false point estimate is not.9293**Triangulate across source *types*, not just documents.** Official data + vendor pricing + user94complaints + hiring activity, all agreeing, is strong. Four blog posts agreeing is one blog post.9596## Step 5 — stopping rules9798Stop when any is true:99- Every sub-question has ≥1 A/B source, and load-bearing ones have two independent.100- New searches return sources you've already seen (**saturation** — the real signal).101- Remaining unknowns wouldn't change the decision (per `thinking/decision-quality`).102- The unknown needs a capability you don't have (browser, paid data, a customer conversation) —103 stop and **name the capability**; that's the finding.104105Anti-pattern: researching until it *feels* thorough. Depth ≠ time spent. The sub-question table106tells you when you're done; feelings don't.107108## Output format109110```111QUESTION112<the decision this serves>113114ANSWER115<2–4 sentences. Confidence stated. Lead with the answer, not the process.>116117SUB-QUESTIONS118| Q | Answer | Grade | Source (url, publisher, pub date, fetched) |119120CONTRADICTIONS121<what disagreed, how resolved, or the unresolved range>122123WHAT I COULD NOT VERIFY124<grade D/E items, failed fetches, tooling gaps — explicitly listed>125126WHAT WOULD CHANGE THE ANSWER127<the specific fetch/experiment that would move it>128```129130The last two sections are **mandatory**. A report without them reads as more certain than it is,131which is the harm this skill exists to prevent.132133## Failure modes134135| Symptom | Cause | Fix |136|---|---|---|137| All sources are blogs | Searched the headline question | Decompose; use site: operators for primary data |138| Confident market size, no N | Number laundered through aggregators | Search the number in quotes; find origin or strike it |139| "No information available" | JS-only pages returned empty | Report tooling gap, don't conclude absence |140| Report is 3,000 words, decision unchanged | No sub-question table; researched for its own sake | Apply stopping rules |141| Sources all agree suspiciously | One origin, many republishers | Trace independence |142| Only positive sources | Never searched the negative | Add `problems`/`alternatives`/`vs` queries |143144## Boundaries145146- Public pages, official APIs, licensed data, public registries only. No login-walled extraction,147 no CAPTCHA/paywall bypass (`sales/outbound-compliance` governs).148- No fabricated citations, ever. No inferred URLs.149- Findings exit via `surface_finding` **with a provenance block** — nothing graded D/E appears in150 a summary field (`thinking/evidence-grading`, hard rule).