Web Research and Monitoring Workflows
Use this umbrella for information retrieval that is not tied to one site-specific API.
Core workflow
- Classify the retrieval need: one-shot search, fresh news, source monitoring, transcript lookup, broad summary, or recurring digest.
- Choose the source path:
- Search APIs/CLIs for broad discovery.
- RSS/Atom/Google News RSS for fresh public news and blocked/rate-limited search.
- Blog watcher feeds for recurring source monitoring.
- YouTube transcript tools for video-content questions.
- Summarizer CLI for known URLs/files.
- Gather more than one source for factual claims unless the user asks for a single-source summary.
- Preserve links, timestamps, publication dates, and provider used.
- For digests, de-duplicate, rank by relevance, and deliver a concise card/report in the requested language.
Labeled playbooks
Search provider selection
Use Brave/Tavily/multi-engine search depending on availability, locality, freshness, and whether AI-optimized snippets or raw web results are needed.
Google News RSS fallback
When normal search is blocked or stale, use Google News RSS, then open the canonical publisher pages for verification.
Blog/RSS monitoring
Store feed lists explicitly, track seen item IDs/URLs, and summarize only new items.
Recurring digest cron jobs
Create self-contained cron prompts with source list, timezone, delivery target, duplicate-job cleanup expectations, and output format.
For scheduled news-card jobs, treat the final response as the delivery payload: do not call messaging tools unless explicitly instructed by a live user. When the prompt asks for a Feishu/Lark-card feel in Markdown, preserve the requested column order and use compact card hierarchy: date title, one-line subtitle, 2-sentence summary, 3 bullets, visual dividers, then each item as title + core content + impact/practical note + inline canonical source link. Avoid AI framing phrases and do not bundle links at the end.
YouTube transcript lookup
Fetch transcript before downloading video. If transcript is missing or incomplete, escalate to the video transcription umbrella.
URL/file summarization
For known documents, prefer direct extraction/summarization rather than a broad web search.
Pitfalls
Never delegate news research to subagents (delegate_task). Subagents systematically hallucinate news stories, dates, headlines, and canonical source URLs. They fabricate plausible-sounding articles that match requested topics — including URLs that look real but return 404. A delegate_task subagent tasked with "find top tech news from the past 24h" will confidently produce fabricated headlines with fabricated URLs across all major publishers (The Verge, Bloomberg, Reuters, BBC, TechCrunch). Always gather and verify news yourself using:
- Direct RSS feeds (BBC, Xinhua, Google News RSS)
- Browser navigation (
browser_navigate()) to trusted publisher homepages
- Google News RSS with canonical URL resolution
- Published API wrappers
If you must parallelize research, use
execute_code with multiple RSS feed fetches rather than delegate_task.
For canonical source URLs in news digests, do not cite Google News RSS links. Open or fetch the publisher page and check that it returns an accessible article URL (HTTP 200 where possible). Some sources block scripted fetches (e.g. Reuters may return 401/403) even when the canonical URL is real; prefer accessible canonical publisher pages when the user explicitly requires reachable links.
Dynamic publisher search pages can hide the final article URL from the static HTML. If browser navigation shows the result but clicking does not route, use browser_console() to inspect anchors, e.g. Array.from(document.querySelectorAll('a')).filter(a => a.textContent.includes('headline fragment')).map(a => a.href), then navigate to the extracted canonical URL.
Don't cite Google News as the underlying source when a canonical publisher URL is available.
Don't infer article contents beyond the headline if the body isn't accessible.
Don't pad weak sections; use a brief version if the day is light.
Reference files
Provider-specific commands, cron prompt examples, and legacy digest formats live in references/from-*.md.
1---2name: web-research-monitoring-workflows3description: Class-level workflow for web search, RSS/news/blog monitoring, source extraction, YouTube transcript lookup, summarization, and recurring news digests. Use when asked to search the web, compare search providers, gather fresh public news under search-rate limits, monitor blogs/RSS, summarize URLs/files/videos, or set up AI/business/news digest cron jobs.4---56# Web Research and Monitoring Workflows78Use this umbrella for information retrieval that is not tied to one site-specific API.910## Core workflow11121. Classify the retrieval need: one-shot search, fresh news, source monitoring, transcript lookup, broad summary, or recurring digest.132. Choose the source path:14 - Search APIs/CLIs for broad discovery.15 - RSS/Atom/Google News RSS for fresh public news and blocked/rate-limited search.16 - Blog watcher feeds for recurring source monitoring.17 - YouTube transcript tools for video-content questions.18 - Summarizer CLI for known URLs/files.193. Gather more than one source for factual claims unless the user asks for a single-source summary.204. Preserve links, timestamps, publication dates, and provider used.215. For digests, de-duplicate, rank by relevance, and deliver a concise card/report in the requested language.2223## Labeled playbooks2425### Search provider selection2627Use Brave/Tavily/multi-engine search depending on availability, locality, freshness, and whether AI-optimized snippets or raw web results are needed.2829### Google News RSS fallback3031When normal search is blocked or stale, use Google News RSS, then open the canonical publisher pages for verification.3233### Blog/RSS monitoring3435Store feed lists explicitly, track seen item IDs/URLs, and summarize only new items.3637### Recurring digest cron jobs3839Create self-contained cron prompts with source list, timezone, delivery target, duplicate-job cleanup expectations, and output format.4041For scheduled news-card jobs, treat the final response as the delivery payload: do not call messaging tools unless explicitly instructed by a live user. When the prompt asks for a Feishu/Lark-card feel in Markdown, preserve the requested column order and use compact card hierarchy: date title, one-line subtitle, 2-sentence summary, 3 bullets, visual dividers, then each item as title + core content + impact/practical note + inline canonical source link. Avoid AI framing phrases and do not bundle links at the end.4243### YouTube transcript lookup4445Fetch transcript before downloading video. If transcript is missing or incomplete, escalate to the video transcription umbrella.4647### URL/file summarization4849For known documents, prefer direct extraction/summarization rather than a broad web search.5051## Pitfalls5253- **Never delegate news research to subagents (delegate_task).** Subagents systematically hallucinate news stories, dates, headlines, and canonical source URLs. They fabricate plausible-sounding articles that match requested topics — including URLs that look real but return 404. A delegate_task subagent tasked with "find top tech news from the past 24h" will confidently produce fabricated headlines with fabricated URLs across all major publishers (The Verge, Bloomberg, Reuters, BBC, TechCrunch). Always gather and verify news yourself using:54 - Direct RSS feeds (BBC, Xinhua, Google News RSS)55 - Browser navigation (`browser_navigate()`) to trusted publisher homepages56 - Google News RSS with canonical URL resolution57 - Published API wrappers58 If you must parallelize research, use `execute_code` with multiple RSS feed fetches rather than `delegate_task`.5960- For canonical source URLs in news digests, do not cite Google News RSS links. Open or fetch the publisher page and check that it returns an accessible article URL (HTTP 200 where possible). Some sources block scripted fetches (e.g. Reuters may return 401/403) even when the canonical URL is real; prefer accessible canonical publisher pages when the user explicitly requires reachable links.6162- Dynamic publisher search pages can hide the final article URL from the static HTML. If browser navigation shows the result but clicking does not route, use `browser_console()` to inspect anchors, e.g. `Array.from(document.querySelectorAll('a')).filter(a => a.textContent.includes('headline fragment')).map(a => a.href)`, then navigate to the extracted canonical URL.6364- Don't cite Google News as the underlying source when a canonical publisher URL is available.65- Don't infer article contents beyond the headline if the body isn't accessible.66- Don't pad weak sections; use a brief version if the day is light.6768## Reference files6970Provider-specific commands, cron prompt examples, and legacy digest formats live in `references/from-*.md`.
davidtoby