Jeremylongshore Claude Code Plugins Plus Skills Firecrawl Performance Tuning

Firecrawl Performance Tuning

Overview

Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.

Latency Benchmarks

Instructions

Step 1: Minimize Formats (Biggest Win)

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// SLOW: requesting everything
const slow = await firecrawl.scrapeUrl(url, {
  formats: ["markdown", "html", "links", "screenshot"],
  // screenshot + full HTML = 3-5x slower
});

// FAST: request only what you need
const fast = await firecrawl.scrapeUrl(url, {
  formats: ["markdown"],   // markdown
  onlyMainContent: true,   // skip nav/footer/sidebar
});

Step 2: Tune waitFor for JS-Heavy Pages

// Default: no JS wait (fastest, works for static sites)
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
  formats: ["markdown"],
  // No waitFor needed — content is in initial HTML
});

// SPA/dynamic pages: add minimal wait
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
  formats: ["markdown"],
  waitFor: 3000,  // 3s — enough for most SPAs
  onlyMainContent: true,
});

// Heavy interactive page: use actions instead of long wait
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
  formats: ["markdown"],
  actions: [
    { type: "wait", selector: ".data-table" },  // wait for specific element
    { type: "scroll", direction: "down" },        // trigger lazy loading
  ],
});

Step 3: Cache Scraped Content

import { LRUCache } from "lru-cache";
import { createHash } from "crypto";

const scrapeCache = new LRUCache<string, any>({
  max: 500,          // max 500 cached pages
  ttl: 3600000,      // 1 hour TTL
});

async function cachedScrape(url: string) {
  const key = createHash("md5").update(url).digest("hex");
  const cached = scrapeCache.get(key);
  if (cached) {
    console.log(`Cache hit: ${url}`);
    return cached;
  }

  const result = await firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
  });

  if (result.success) {
    scrapeCache.set(key, result);
  }
  return result;
}
// Typical savings: 50-80% credit reduction for repeated scrapes

Step 4: Use Batch Scrape for Multiple URLs

// SLOW: sequential individual scrapes
const urls = ["https://a.com", "https://b.com", "https://c.com"];
for (const url of urls) {
  await firecrawl.scrapeUrl(url, { formats: ["markdown"] }); // 3 API calls
}

// FAST: single batch scrape call
const batchResult = await firecrawl.batchScrapeUrls(urls, {
  formats: ["markdown"],
  onlyMainContent: true,
});
// 1 API call, internally parallelized

Step 5: Map Before Crawl (Save Credits)

// EXPENSIVE: crawl everything, filter later
await firecrawl.crawlUrl("https://docs.example.com", { limit: 1000 });

// CHEAPER: map first (1 credit), then scrape only what you need
const map = await firecrawl.mapUrl("https://docs.example.com");
const apiDocs = (map.links || []).filter(url =>
  url.includes("/api/") || url.includes("/reference/")
);
console.log(`Map: ${map.links?.length} total, ${apiDocs.length} relevant`);

// Batch scrape only relevant URLs
const result = await firecrawl.batchScrapeUrls(apiDocs.slice(0, 50), {
  formats: ["markdown"],
});

Step 6: Measure Scrape Performance

async function timedScrape(url: string) {
  const start = Date.now();
  const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
  const duration = Date.now() - start;

  console.log({
    url,
    durationMs: duration,
    contentLength: result.markdown?.length || 0,
    success: result.success,
    charsPerSecond: Math.round((result.markdown?.length || 0) / (duration / 1000)),
  });

  return result;
}

Error Handling

Examples

Performance Comparison Script

const url = "https://docs.firecrawl.dev";

// Compare different format configurations
for (const formats of [["markdown"], ["markdown", "html"], ["markdown", "html", "screenshot"]]) {
  const start = Date.now();
  await firecrawl.scrapeUrl(url, { formats: formats as any, onlyMainContent: true });
  console.log(`${formats.join(",")}: ${Date.now() - start}ms`);
}

Resources

• Advanced Scraping Guide
• Batch Scrape
• Map Endpoint
• LRU Cache

Next Steps

For cost optimization, see firecrawl-cost-tuning.

Converted and distributed by TomeVault — claim your Tome and manage your conversions.