436 Firecrawl Reference Architecture Bec89b62

firecrawl-reference-architecture

Skill Scaffold

firecrawl-reference-architecture/
  SKILL.md

File Descriptions

1. SKILL.md

Purpose: Implement production-ready reference architecture with data pipeline layers, storage integration, and processing components. Workflow: Architecture skill - establishes team-wide patterns for FireCrawl-based data pipelines. Relates to: Synthesizes patterns from all operational skills; foundational for firecrawl-architecture-variants

Summary

This skill implements a production-ready reference architecture for FireCrawl data pipelines. It covers the ingestion layer (URL sources, scheduling, deduplication), scraping layer (FireCrawl client wrapper, error handling, rate limiting), processing layer (markdown cleanup, content extraction, transformation), storage layer (database integration, file storage, vector databases), and monitoring layer (metrics, logging, alerting). This architecture serves as the foundation for team-wide scraping infrastructure.