firecrawl-batch-processing
Skill Scaffold
firecrawl-batch-processing/
SKILL.md
File Descriptions
1. SKILL.md
Purpose: Implement batch URL processing with parallel execution, progress tracking, result aggregation, and failure handling. Workflow: High-volume skill - enables processing of large URL lists with optimal throughput and reliability. Relates to: Builds on firecrawl-rate-limits; extends firecrawl-crawl-site for custom URL lists
Summary
This skill implements efficient batch processing for large URL lists with FireCrawl. It covers parallel request execution with concurrency limits, progress tracking and reporting, result aggregation into structured datasets, failure handling with retry queues, and checkpoint/resume capability for long-running batches. This enables processing of thousands of URLs efficiently while respecting rate limits and handling failures gracefully.