Firecrawl Cost Tuning

Firecrawl Cost Tuning

Overview

Reduce Firecrawl web scraping costs by limiting crawl depth, caching scraped content, and choosing the right scrape mode. Firecrawl charges credits per page: 1 credit for single-page scrape, variable credits for crawl jobs (each discovered page costs credits).

Prerequisites

• Firecrawl account with credit balance visibility
• Understanding of your scraping patterns (single page vs crawl)
• Cache infrastructure for storing scraped content

Instructions

Step 1: Always Set Crawl Limits

set -euo pipefail
# BAD: Unbounded crawl (could consume thousands of credits)
curl -X POST https://api.firecrawl.dev/v1/crawl \
  -d '{"url": "https://docs.example.com"}'

# GOOD: Bounded crawl with depth and page limits
curl -X POST https://api.firecrawl.dev/v1/crawl \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 2,
    "limit": 50,
    "scrapeOptions": {"formats": ["markdown"]}
  }'
# Saves: potentially hundreds or thousands of credits per crawl

Step 2: Use Scrape for Known URLs Instead of Crawl

// If you know the specific pages you need, scrape individually instead of crawling
const targetUrls = [
  'https://docs.example.com/api/authentication',
  'https://docs.example.com/api/endpoints',
  'https://docs.example.com/api/errors',
];

// Scrape 3 specific pages: 3 credits
for (const url of targetUrls) {
  await firecrawl.scrapeUrl(url, { formats: ['markdown'] });
}
// vs. Crawl entire docs site: potentially 500+ credits  # HTTP 500 Internal Server Error

Step 3: Cache Scraped Content

import { createHash } from 'crypto';

const scrapeCache = new Map<string, { content: string; timestamp: number }>();
const CACHE_TTL = 24 * 3600 * 1000; // 24 hours  # 1000: 3600: timeout: 1 hour

async function cachedScrape(url: string): Promise<string> {
  const key = createHash('md5').update(url).digest('hex');
  const cached = scrapeCache.get(key);
  if (cached && Date.now() - cached.timestamp < CACHE_TTL) return cached.content;

  const result = await firecrawl.scrapeUrl(url, { formats: ['markdown'] });
  scrapeCache.set(key, { content: result.markdown, timestamp: Date.now() });
  return result.markdown;
}
// Typical savings: 50-80% credit reduction for recurring scrape patterns

Step 4: Choose Minimal Scrape Options

set -euo pipefail
# Only request what you need -- each format option has cost implications
# Minimal (cheapest): markdown only
curl -X POST https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

# Avoid requesting screenshots, PDFs, or rawHtml unless actually needed
# Each additional format increases processing time and may affect credit usage

Step 5: Monitor Credit Efficiency

set -euo pipefail
# Find which crawl jobs consumed the most credits
curl -s https://api.firecrawl.dev/v1/usage \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" | \
  jq '{
    credits_remaining: .credits_remaining,
    credits_used_today: .credits_used_today,
    avg_credits_per_job: (.credits_used_month / (.jobs_count + 0.01)),
    projected_monthly: (.credits_used_today * 30)
  }'

Error Handling

Examples

Basic usage: Apply firecrawl cost tuning to a standard project setup with default configuration options.

Advanced scenario: Customize firecrawl cost tuning for production environments with multiple constraints and team-specific requirements.

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale

Resources

• Official monitoring documentation
• Community best practices and patterns
• Related skills in this plugin pack