Firecrawl Architecture Variants

FireCrawl Architecture Variants

Overview

Three validated architecture blueprints for FireCrawl integrations.

Prerequisites

• Understanding of team size and DAU requirements
• Knowledge of deployment infrastructure
• Clear SLA requirements
• Growth projections available

Variant A: Monolith (Simple)

Best for: MVPs, small teams, < 10K daily active users

my-app/
├── src/
│   ├── firecrawl/
│   │   ├── client.ts          # Singleton client
│   │   ├── types.ts           # Types
│   │   └── middleware.ts      # Express middleware
│   ├── routes/
│   │   └── api/
│   │       └── firecrawl.ts    # API routes
│   └── index.ts
├── tests/
│   └── firecrawl.test.ts
└── package.json

Key Characteristics

• Single deployment unit
• Synchronous FireCrawl calls in request path
• In-memory caching
• Simple error handling

Code Pattern

// Direct integration in route handler
app.post('/api/create', async (req, res) => {
  try {
    const result = await firecrawlClient.create(req.body);
    res.json(result);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Variant B: Service Layer (Moderate)

Best for: Growing startups, 10K-100K DAU, multiple integrations

my-app/
├── src/
│   ├── services/
│   │   ├── firecrawl/
│   │   │   ├── client.ts      # Client wrapper
│   │   │   ├── service.ts     # Business logic
│   │   │   ├── repository.ts  # Data access
│   │   │   └── types.ts
│   │   └── index.ts           # Service exports
│   ├── controllers/
│   │   └── firecrawl.ts
│   ├── routes/
│   ├── middleware/
│   ├── queue/
│   │   └── firecrawl-processor.ts  # Async processing
│   └── index.ts
├── config/
│   └── firecrawl/
└── package.json

Key Characteristics

• Separation of concerns
• Background job processing
• Redis caching
• Circuit breaker pattern
• Structured error handling

Code Pattern

// Service layer abstraction
class FireCrawlService {
  constructor(
    private client: FireCrawlClient,
    private cache: CacheService,
    private queue: QueueService
  ) {}

  async createResource(data: CreateInput): Promise<Resource> {
    // Business logic before API call
    const validated = this.validate(data);

    // Check cache
    const cached = await this.cache.get(cacheKey);
    if (cached) return cached;

    // API call with retry
    const result = await this.withRetry(() =>
      this.client.create(validated)
    );

    // Cache result
    await this.cache.set(cacheKey, result, 300);

    // Async follow-up
    await this.queue.enqueue('firecrawl.post-create', result);

    return result;
  }
}

Variant C: Microservice (Complex)

Best for: Enterprise, 100K+ DAU, strict SLAs

firecrawl-service/              # Dedicated microservice
├── src/
│   ├── api/
│   │   ├── grpc/
│   │   │   └── firecrawl.proto
│   │   └── rest/
│   │       └── routes.ts
│   ├── domain/
│   │   ├── entities/
│   │   ├── events/
│   │   └── services/
│   ├── infrastructure/
│   │   ├── firecrawl/
│   │   │   ├── client.ts
│   │   │   ├── mapper.ts
│   │   │   └── circuit-breaker.ts
│   │   ├── cache/
│   │   ├── queue/
│   │   └── database/
│   └── index.ts
├── config/
├── k8s/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── hpa.yaml
└── package.json

other-services/
├── order-service/       # Calls firecrawl-service
├── payment-service/
└── notification-service/

Key Characteristics

• Dedicated FireCrawl microservice
• gRPC for internal communication
• Event-driven architecture
• Database per service
• Kubernetes autoscaling
• Distributed tracing
• Circuit breaker per service

Code Pattern

// Event-driven with domain isolation
class FireCrawlAggregate {
  private events: DomainEvent[] = [];

  process(command: FireCrawlCommand): void {
    // Domain logic
    const result = this.execute(command);

    // Emit domain event
    this.events.push(new FireCrawlProcessedEvent(result));
  }

  getUncommittedEvents(): DomainEvent[] {
    return [...this.events];
  }
}

// Event handler
@EventHandler(FireCrawlProcessedEvent)
class FireCrawlEventHandler {
  async handle(event: FireCrawlProcessedEvent): Promise<void> {
    // Saga orchestration
    await this.sagaOrchestrator.continue(event);
  }
}

Decision Matrix

Migration Path

Monolith → Service Layer:
1. Extract FireCrawl code to service/
2. Add caching layer
3. Add background processing

Service Layer → Microservice:
1. Create dedicated firecrawl-service repo
2. Define gRPC contract
3. Add event bus
4. Deploy to Kubernetes
5. Migrate traffic gradually

Instructions

Step 1: Assess Requirements

Use the decision matrix to identify appropriate variant.

Step 2: Choose Architecture

Select Monolith, Service Layer, or Microservice based on needs.

Step 3: Implement Structure

Set up project layout following the chosen blueprint.

Step 4: Plan Migration Path

Document upgrade path for future scaling.

Output

• Architecture variant selected
• Project structure implemented
• Migration path documented
• Appropriate patterns applied

Error Handling

Examples

Quick Variant Check

# Count team size and DAU to select variant
echo "Team: $(git log --format='%ae' | sort -u | wc -l) developers"
echo "DAU: Check analytics dashboard"

Resources

• Monolith First
• Microservices Guide
• FireCrawl Architecture Guide

Next Steps

For common anti-patterns, see firecrawl-known-pitfalls.