Skill Scaffold

This skill implements efficient batch processing for large URL lists with FireCrawl.

tools-only Updated 7 repo stars

File contents

firecrawl-batch-processing

Skill Scaffold

firecrawl-batch-processing/
  SKILL.md

File Descriptions

1. SKILL.md

Purpose: Implement batch URL processing with parallel execution, progress tracking, result aggregation, and failure handling. Workflow: High-volume skill - enables processing of large URL lists with optimal throughput and reliability. Relates to: Builds on firecrawl-rate-limits; extends firecrawl-crawl-site for custom URL lists

Summary

This skill implements efficient batch processing for large URL lists with FireCrawl. It covers parallel request execution with concurrency limits, progress tracking and reporting, result aggregation into structured datasets, failure handling with retry queues, and checkpoint/resume capability for long-running batches. This enables processing of thousands of URLs efficiently while respecting rate limits and handling failures gracefully.

tools-only/X-Skills/tree/main/automation/scripting/566-firecrawl-batch-processing_e55ee7ef commit 2e1d4d871e

Frequently asked questions

npx skillmds@latest add tools-only/skill-scaffold-54