Web Research Fanout

Web Research Fanout

Core principle: coverage comes from independent sources, not from the number of agents. The lane count equals the number of genuinely independent lines (sub-question × source type), not a fixed number you feel like dispatching. N researchers asked the same question will pick the same pages from the same search results → correlated errors reinforce each other and produce "confident mistakes".

0. First: decide whether to fan out

• Single fact (one number, one link) → dispatch nothing, or one agent; the main session searches directly.
• You need "a map" (breadth, multi-source corroboration, conflicting claims) → fan out.
• Reading local documents/PDFs → use dual-read, not this skill.

1. Firepower tiers (lane count by stake, not by mood)

Lane examples (cut by source type): official announcements/docs | code repos & issues | benchmarks/review sites | community reports (Reddit/HN/V2EX) | Chinese sources | pricing pages/terms | regulators/primary documents. Every lane's researcher must get a different seed (wording / language / site restriction — at least one must differ), otherwise it's a clone.

Quality formula: quality = independent sources × adversarial verification × arbitration; raw numbers are just an amplifier. Before adding a lane, ask: in which dimension is it independent from the existing ones — source type, language, stance, model family? If none, it's a clone and will only amplify correlated error. Model assignment: lanes are standardized work — mid-tier models are enough (in practice: contract compliance is fine; failures happen in the channels, not the thinking). Judgment-heavy steps are only three: the main session's planning/merging, the adversarial lane, and arbitration. Different family ≠ more expensive: the point of cross-family is uncorrelated blind spots; any mid-tier model from a different family will do.

2. Dispatch (one message, parallel; batch if heavy)

• Send multiple Agent calls (subagent_type: researcher) in one message — do not serialize.
• ≤6 per batch; heavy tier dispatches in batches (avoid gateway rate limits).
• Validate channels before writing them into tasks: the main session verifies one working channel first, then writes the complete command into the task template — sub-agents have no Skill tool and cannot load other manuals, so commands must be self-contained. Common channels: search APIs (Exa etc., via mcporter/MCP or curl) | search-engine result pages (Bing/Brave) | r.jina.ai for page reading | gh | yt-dlp; verify social-platform channels before using them.
• Every task must be self-contained (sub-agents cannot see the main session). Template:

Research task (lane: <name>).
Goal: <one-sentence question>. Scope: <source type / sites / language>. Seeds: <3-5 keywords or site restrictions>.
Tools: prefer Bash with your search command, e.g. mcporter call 'exa.web_search_exa(query: "...", numResults: 8)' (replace with your environment's);
fetch pages with node <pack-root>/skills/web-research-fanout/scripts/fetch-hard.mjs <url> (chained degradation: real-UA direct fetch → HTML-to-text → r.jina fallback) or WebFetch;
search-result-page fallback (use only if the two above fail): www.bing.com/search?q= / search.brave.com/search?q=; DuckDuckGo is CAPTCHA-walled, don't use it;
on 403/blocks: escalate one level (see §6); if still blocked, stop and mark "blocked: <site> <status code>" — never force it.
Output contract (strict):
1) claims table, one row per claim: assertion | source URL | date | primary/secondary | counter-evidence
2) "not found / conflicting" list
3) ≤3-line summary
Nothing else. Budget: ≤8 searches + ≤10 page fetches; stop once a claim has 2 independent sources.

Big topics (heavy tier / decision-grade conclusions): two rounds, don't one-shot

1. Recon round (main session, 5-10 min): search 3-6 times yourself to learn the key entities, where the primary sources live, and where accounts disagree → use that to cut lanes and seeds. (For high-cost tasks you may add a "plan gate": show the plan to the user before starting.)
2. Collection round: dispatch researchers per lane in parallel (≤6 per batch).
3. Gap round: dispatch 1-2 targeted researchers for exactly three things — the "not found" list, conflicts, and counter-evidence for load-bearing claims.

The cost of one-shotting: lanes burn their budget guessing keywords (measured).

3. Mechanical merge in the main session (don't read every report)

• Deduplicate by assertion: same assertion from multiple sources → one row, note the source count.
• Mechanically flag two things: ① load-bearing assertions with a single source ② assertions with conflicting conclusions → into the "conflict list".
• Keep only the merged table + conflict list; never stuff all raw reports into context.

4. Verification & arbitration (default: primary sources + adversarial + cross-family)

For every load-bearing assertion (an assertion the conclusion rests on), do all three by default at standard tier and above; quick tier does at least ①:

1. Primary-source trace: the assertion must be traceable to a primary source (official page / official repo / original document). Secondary-only → mark "pending primary verification" and do not treat as settled.
2. Adversarial lane: dispatch one researcher whose job is to find counter-evidence — search "X is wrong / not recommended / complaints / scam", find primary documents that contradict. Finding none is itself a result (write one line: "no counter-evidence found").
3. Cross-family arbitration: the verifier/arbiter uses a model from a different family than the lanes — the point is uncorrelated blind spots, not a smarter model. Feed it only the conflict list + adversarial results, and require verbatim quotes from the originals as grounds.

• Arbitration output uses the match/addition/conflict tri-classification (same as dual-read) + for each conflict: each side's sources, dates, primacy, recommended resolution or "disputed".
• Majority vote ≠ truth: independent-source count > vote count; primary > secondary; newer > older.

5. Output format

Conclusions table (assertion | sources | date | confidence) + disputed list (with both sides and their sources) + a one-sentence summary. Disputed items must be explicitly surfaced — never silently dropped.

Pre-delivery self-check (all must pass):

• What evidence would overturn this conclusion? Where is the adversarial lane's output row?
• Do all load-bearing assertions trace to primary sources? Are secondary-only ones marked "pending primary verification"?
• Freshness: key numbers ≤7 days, background ≤30 days; flag anything older as "stale" with a note.
• Is the "not found" list preserved verbatim? Was anything quietly deleted?

6. Anti-scrape escalation ladder (escalate in order; never retry the same rung)

On any 403/block: go up one level; do not retry the same level. Retrying is meaningless against a real block — measured: Reddit returned 403 to plain curl+UA, headless browsers, r.jina, and WebFetch alike = exit-IP-level ban; no stronger scraper gets in.

• Researchers may only use L0-L2; escalate at most one level; if still blocked → stop and mark blocked: <site> <status> in the lane report, hand it to the main session. Never burn budget forcing it.
• Channels differ per site (measured): fetch-hard succeeds on ordinary sites; on hard sites (Reddit, Zhihu) direct fetch and r.jina both return 403 — jina only works on ordinary sites. The cure for hard sites is the L1 index layer, not more effort at L2.
• L1 returns an index snapshot (may be stale): verify key assertions via L0/L2, or mark the date column as "index time".
• Sub-agents usually have no browser tools — hard sites always go "main session fetches → saves to disk → sub-agents read".

Appendix: budget and known pitfalls (measured)

• Real cost (4-lane low-end run; the standard tier is 5-8 lanes): per lane ≈84K-262K tokens (depends on pages fetched); 4 lanes ≈560K tokens total, ≈5 min wall clock (parallel) + ≈1 min arbitration. Heavy tier (10-20 lanes): ≈1.5M-4M tokens per round — check your quota before opening it.
• Premium cost (adversarial lane + cross-family arbitration + two rounds): ≈1.5-2.5× standard, 15-25 min wall clock; worth it when the conclusion is decision-grade.
• Lane productivity ordering (measured): official docs/primary pages > GitHub READMEs/issues > community threads. EN/ZH community lanes cost the most and produce the least.
• Reddit-class hard sites may be blocked at the exit-IP level (curl+UA, headless browser, r.jina, WebFetch all blocked) → use the L1 index layer (readable highlights, measured); for HN use the Algolia API (hn.algolia.com/api/v1/search?query=..., never blocked); if neither works, mark "not obtained" — don't force it.
• Bing's result page returns garbage for Chinese queries (tokenization failure) → use Brave or English seeds for Chinese lanes.
• "Free tier" numbers tend to hide behind 403'd pricing pages → needs Bash+curl or a logged-in channel; if unavailable, mark "disputed: no primary source obtained".
• Failure and timeout handling: lane dispatch failures/timeouts (killed for 10 min of inactivity / turn execution failed / stopped) → retry once (smaller scope or different model); still failing → record it in the gap list. Never skip silently. Lane fetches go through fetch-hard (which carries its own timeouts) so nothing hangs.
• Channel ranking (measured): search API (mcporter/Exa) > fetch-hard (L2) > Brave/Bing result-page scraping (quality varies) > main-session browser (L4).
• Sub-agent configs are usually session-start snapshots in many hosts: changing a role's tools/system prompt requires a new session.
• This protocol borrowed ideas from contemporary open-source projects (red-team/claims-ledger, review isolation, etc.) — see the repo README's Related work.