Firecrawl Migration Deep Dive

FireCrawl Migration Deep Dive

Overview

Comprehensive guide for migrating to or from FireCrawl, or major version upgrades.

Prerequisites

• Current system documentation
• FireCrawl SDK installed
• Feature flag infrastructure
• Rollback strategy tested

Migration Types

Pre-Migration Assessment

Step 1: Current State Analysis

set -euo pipefail
# Document current implementation
find . -name "*.ts" -o -name "*.py" | xargs grep -l "firecrawl" > firecrawl-files.txt

# Count integration points
wc -l firecrawl-files.txt

# Identify dependencies
npm list | grep firecrawl
pip freeze | grep firecrawl

Step 2: Data Inventory

interface MigrationInventory {
  dataTypes: string[];
  recordCounts: Record<string, number>;
  dependencies: string[];
  integrationPoints: string[];
  customizations: string[];
}

async function assessFireCrawlMigration(): Promise<MigrationInventory> {
  return {
    dataTypes: await getDataTypes(),
    recordCounts: await getRecordCounts(),
    dependencies: await analyzeDependencies(),
    integrationPoints: await findIntegrationPoints(),
    customizations: await documentCustomizations(),
  };
}

Migration Strategy: Strangler Fig Pattern

Phase 1: Parallel Run
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   System    │ ──▶ │  FireCrawl   │
│   (100%)    │     │   (0%)      │
└─────────────┘     └─────────────┘

Phase 2: Gradual Shift
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   (50%)     │ ──▶ │   (50%)     │
└─────────────┘     └─────────────┘

Phase 3: Complete
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   (0%)      │ ──▶ │   (100%)    │
└─────────────┘     └─────────────┘

Implementation Plan

Phase 1: Setup (Week 1-2)

set -euo pipefail
# Install FireCrawl SDK
npm install @firecrawl/sdk

# Configure credentials
cp .env.example .env.firecrawl
# Edit with new credentials

# Verify connectivity
node -e "require('@firecrawl/sdk').ping()"

Phase 2: Adapter Layer (Week 3-4)

// src/adapters/firecrawl.ts
interface ServiceAdapter {
  create(data: CreateInput): Promise<Resource>;
  read(id: string): Promise<Resource>;
  update(id: string, data: UpdateInput): Promise<Resource>;
  delete(id: string): Promise<void>;
}

class FireCrawlAdapter implements ServiceAdapter {
  async create(data: CreateInput): Promise<Resource> {
    const firecrawlData = this.transform(data);
    return firecrawlClient.create(firecrawlData);
  }

  private transform(data: CreateInput): FireCrawlInput {
    // Map from old format to FireCrawl format
  }
}

Phase 3: Data Migration (Week 5-6)

async function migrateFireCrawlData(): Promise<MigrationResult> {
  const batchSize = 100;
  let processed = 0;
  let errors: MigrationError[] = [];

  for await (const batch of oldSystem.iterateBatches(batchSize)) {
    try {
      const transformed = batch.map(transform);
      await firecrawlClient.batchCreate(transformed);
      processed += batch.length;
    } catch (error) {
      errors.push({ batch, error });
    }

    // Progress update
    console.log(`Migrated ${processed} records`);
  }

  return { processed, errors };
}

Phase 4: Traffic Shift (Week 7-8)

// Feature flag controlled traffic split
function getServiceAdapter(): ServiceAdapter {
  const firecrawlPercentage = getFeatureFlag('firecrawl_migration_percentage');

  if (Math.random() * 100 < firecrawlPercentage) {
    return new FireCrawlAdapter();
  }

  return new LegacyAdapter();
}

Rollback Plan

set -euo pipefail
# Immediate rollback
kubectl set env deployment/app FIRECRAWL_ENABLED=false
kubectl rollout restart deployment/app

# Data rollback (if needed)
./scripts/restore-from-backup.sh --date YYYY-MM-DD

# Verify rollback
curl https://app.yourcompany.com/health | jq '.services.firecrawl'

Post-Migration Validation

async function validateFireCrawlMigration(): Promise<ValidationReport> {
  const checks = [
    { name: 'Data count match', fn: checkDataCounts },
    { name: 'API functionality', fn: checkApiFunctionality },
    { name: 'Performance baseline', fn: checkPerformance },
    { name: 'Error rates', fn: checkErrorRates },
  ];

  const results = await Promise.all(
    checks.map(async c => ({ name: c.name, result: await c.fn() }))
  );

  return { checks: results, passed: results.every(r => r.result.success) };
}

Instructions

Assess current configuration

Document existing implementation and data inventory.

Step 2: Build Adapter Layer

Create abstraction layer for gradual migration.

Step 3: Migrate Data

Run batch data migration with error handling.

Step 4: Shift Traffic

Gradually route traffic to new FireCrawl integration.

Output

• Migration assessment complete
• Adapter layer implemented
• Data migrated successfully
• Traffic fully shifted to FireCrawl

Error Handling

Examples

Quick Migration Status

const status = await validateFireCrawlMigration();
console.log(`Migration ${status.passed ? 'PASSED' : 'FAILED'}`);
status.checks.forEach(c => console.log(`  ${c.name}: ${c.result.success}`));

Resources

• Strangler Fig Pattern
• FireCrawl Migration Guide

Flagship+ Skills

For advanced troubleshooting, see firecrawl-advanced-troubleshooting.