Web Scraper

Web Scraper — Web Scraping System Construction Pipeline

An agent team collaborates to build target analysis, crawler design, parsing, storage, and monitoring for web scraping systems.

Execution Mode

Agent Team — Five agents communicate directly via SendMessage and perform cross-validation.

Agent Composition

Workflow

Phase 1: Preparation (performed directly by the orchestrator)

1. Extract the following from user input:
   • Target site URL: Website to scrape
   • Target data: What data to extract
   • Purpose: Intended use of collected data (analysis/monitoring/archiving)
   • Scale: Expected data volume, collection frequency
   • Constraints (optional): Tech stack limitations, budget, legal requirements
2. Create the _workspace/ directory at the project root
3. Organize the input and save it to _workspace/00_input.md
4. Create the _workspace/src/ directory
5. If pre-existing files are available, copy them to _workspace/ and skip the corresponding phase
6. Determine the execution mode based on the scope of the request (see "Execution Modes" below)

Phase 2: Team Assembly and Execution

Assemble the team and assign tasks. Inter-task dependencies are as follows:

Tasks 2a (crawler) and 2b (parser) run in parallel since both depend only on Task 1 (analysis).

Inter-agent communication flow:

• analyst completes > passes URL patterns, anti-bot info, rate limits to crawler; data points and DOM structure to parser
• crawler completes > passes raw data format to parser; crawler health checkpoints to monitor
• parser completes > passes data schema to data-mgr; parsing metrics to monitor
• data-mgr completes > passes data quality metrics to monitor
• monitor integrates all components to finalize operations configuration

Phase 3: Integration and Final Deliverables

1. Verify all files in _workspace/ and _workspace/src/
2. Validate cross-deliverable consistency (analysis > crawler > parser > storage > monitoring)
3. Present the final summary and execution instructions to the user

Execution Modes by Request Scope

Reusing existing files: If the user provides existing analysis results or crawler code, copy to _workspace/ and skip the corresponding agent.

Data Transfer Protocol

Error Handling

Test Scenarios

Normal Flow

Prompt: "Build a scraper to collect product prices from this e-commerce site daily" Expected result:

• Analysis: Site structure, pagination, anti-bot mechanisms, robots.txt compliance plan
• Crawler: Async httpx-based crawler with rate limiting and retry logic
• Parser: CSS selector-based price/title/URL extraction with validation
• Storage: SQLite with upsert deduplication, CSV daily export
• Monitoring: Cron schedule, parsing success rate alerts, site change detection

Analysis-Only Flow

Prompt: "Analyze whether this site can be scraped" Expected result:

• target-analyst performs full analysis and risk assessment
• Other agents are not deployed

Error Flow

Prompt: "Scrape data from this SPA with Cloudflare protection" Expected result:

• target-analyst identifies Cloudflare challenge and SPA rendering
• crawler-developer uses Playwright with appropriate wait strategies
• parser-engineer handles dynamic DOM with robust selectors
• monitor-operator sets up change detection for frequently updated selectors

Agent Extension Skills