Anti Scraping

Anti-Scraping & Web Scraping

When to use: Websites with Cloudflare protection, JavaScript rendering requirements, or anti-bot measures.

Overview

Provides battle-tested solutions for bypassing common anti-scraping measures using Playwright headless browser with stealth configurations.

Key Capabilities

• ✅ Cloudflare challenge bypass
• ✅ JavaScript rendering
• ✅ Real browser context simulation
• ✅ Stealth mode (hides automation detection)
• ✅ Screenshot capture for debugging

Quick Start

Prerequisites

# Install Playwright
npm install -g playwright
playwright install chromium

Basic Usage Pattern

// n8n Execute Command node
const { execSync } = require('child_process');

const url = 'https://example.com';
const outputFile = '/tmp/page.html';

// Playwright command with stealth
const command = `node playwright-cloudflare.js "${url}" "${outputFile}"`;
execSync(command);

// Read result
const html = fs.readFileSync(outputFile, 'utf8');

Core Script: playwright-cloudflare.js

Location: n8n-skills/anti-scraping/playwright-cloudflare.js

Key Features:

• Disables automation detection
• Sets real browser headers
• Configures viewport and user agent
• Handles Cloudflare waiting
• Captures screenshots on failure

Configuration:

const config = {
  waitForCloudflare: true,      // Wait for CF challenge
  waitTime: 15000,               // Max wait time (ms)
  selector: '.product-list',     // Element to wait for
  screenshotOnError: true,       // Debug screenshots
  userAgent: 'Mozilla/5.0...'   // Real browser UA
};

n8n Workflow Pattern

[Manual Trigger]
    ↓
[Set Parameters]
    target_url: https://site.com
    wait_selector: .content
    ↓
[Execute Command: Playwright]
    Command: node
    Arguments: playwright-cloudflare.js {{$json.target_url}} /tmp/output.html
    ↓
[Read HTML File]
    File: /tmp/output.html
    ↓
[Parse with Cheerio]
    (use html-parsing skill)

Performance

• Speed: 15-25 seconds per page
• Success Rate: ~95% for Cloudflare sites
• Resource Usage: ~200-300MB RAM per browser instance

Troubleshooting

Cloudflare Still Blocking

# Increase wait time
--wait 30000

# Add specific selector to wait for
--selector '.product-list'

# Check screenshot for errors
/tmp/error-screenshot.png

Timeout Errors

# Increase timeout in playwright script
timeout: 60000  // 60 seconds

Memory Issues

# Close browser properly
await browser.close();

# Limit concurrent instances
# Use n8n Split Into Batches with batch size = 1

Best Practices

1. Add Delays: Wait 3-5 seconds between requests
2. Rotate User Agents: Change UA periodically
3. Use Residential Proxies: For high-volume scraping
4. Handle Errors: Implement retry logic with exponential backoff
5. Respect robots.txt: Check site policies

Common Patterns

Pattern 1: Single Page Scraping

Trigger → Playwright → Parse → Export

Pattern 2: Multi-Page with Pagination

Trigger → Generate URLs (pagination skill) →
Split Into Batches → Playwright → Wait 5s →
Parse → Deduplicate → Export

Pattern 3: With Error Handling

Playwright → [Error Trigger] → Retry Logic → Notification

Integration with Other Skills

• pagination: Generate URLs for multi-page scraping
• html-parsing: Extract data from rendered HTML
• error-handling: Retry on failures
• debugging: Validate extracted data

Full Code and Documentation

Complete implementation with examples: /mnt/d/work/n8n_agent/n8n-skills/anti-scraping/

Files:

• playwright-cloudflare.js - Main scraping script
• README.md - Detailed documentation
• example-workflow.json - n8n workflow example
• config.template.env - Configuration template