Puppeteer Browser Automation

Puppeteer Browser Automation

This skill makes an AI agent write reliable Puppeteer scripts: launching Chrome with the right flags, navigating and interacting without race conditions, capturing screenshots and PDFs, and intercepting network requests to mock or block traffic. Trigger it when a project uses puppeteer or puppeteer-core, or when the user asks to scrape a page, generate a PDF from HTML, screenshot a site, or automate Chrome without a full test framework.

Core Principles

1. Every action must wait for its precondition. page.click() immediately after page.goto() races against rendering. Use page.locator() (auto-waiting, Puppeteer 21+) or explicit waitForSelector before every interaction.
2. Never use fixed sleeps. await new Promise(r => setTimeout(r, 3000)) is either too short (flaky) or too long (slow). Wait on selectors, network idle, or response predicates instead.
3. Always close the browser in finally. A script that throws before browser.close() leaks a Chrome process. In CI those zombies accumulate until the runner dies.
4. Set waitUntil deliberately. load waits for every image and font; domcontentloaded is enough for interaction; networkidle2 is for SPAs that fetch after load. Pick per page, do not cargo-cult networkidle0.
5. Combine navigation-triggering actions with Promise.all. Clicking a link then awaiting waitForNavigation separately misses fast navigations. Start the wait before the click.
6. Request interception is your mock layer. Block analytics and images for speed, stub API responses for determinism — no proxy server needed.

Setup

npm install --save-dev puppeteer typescript tsx

// src/browser.ts
import puppeteer, { Browser } from 'puppeteer';

export async function launchBrowser(): Promise<Browser> {
  return puppeteer.launch({
    headless: true,
    args: [
      '--no-sandbox', // required in most Docker/CI containers
      '--disable-dev-shm-usage', // /dev/shm is 64MB in Docker; avoids renderer crashes
      '--disable-gpu',
      '--window-size=1366,768',
    ],
    defaultViewport: { width: 1366, height: 768 },
  });
}

A complete script with correct lifecycle handling:

// src/check-login.ts
import { launchBrowser } from './browser';

async function main(): Promise<void> {
  const browser = await launchBrowser();
  try {
    const page = await browser.newPage();
    page.setDefaultTimeout(15_000);

    await page.goto('https://practice.expandtesting.com/login', {
      waitUntil: 'domcontentloaded',
    });

    // locator() auto-waits for visibility and stability before acting
    await page.locator('#username').fill('practice');
    await page.locator('#password').fill('SuperSecretPassword!');

    await Promise.all([
      page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
      page.locator('button[type="submit"]').click(),
    ]);

    const flash = await page.locator('#flash').waitHandle();
    const text = await flash.evaluate((el) => el.textContent?.trim());
    if (!text?.includes('You logged into a secure area')) {
      throw new Error(`Login failed, flash message: ${text}`);
    }
    console.log('Login OK');
  } finally {
    await browser.close();
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

npx tsx src/check-login.ts

Patterns

Screenshots and PDF Generation

import { launchBrowser } from './browser';

const browser = await launchBrowser();
try {
  const page = await browser.newPage();
  await page.goto('https://qaskills.sh', { waitUntil: 'networkidle2' });

  // Full-page screenshot
  await page.screenshot({ path: 'homepage.png', fullPage: true });

  // Screenshot of one element only
  const hero = await page.waitForSelector('main section:first-of-type');
  await hero!.screenshot({ path: 'hero.png' });

  // PDF requires headless mode; emulate print CSS first
  await page.emulateMediaType('print');
  await page.pdf({
    path: 'homepage.pdf',
    format: 'A4',
    printBackground: true,
    margin: { top: '20mm', bottom: '20mm', left: '15mm', right: '15mm' },
  });
} finally {
  await browser.close();
}

Request Interception: Block Noise, Stub APIs

const page = await browser.newPage();
await page.setRequestInterception(true);

page.on('request', (request) => {
  const url = request.url();
  const type = request.resourceType();

  // Block images, fonts, and trackers for a 3-5x speedup on content scraping
  if (type === 'image' || type === 'font' || url.includes('google-analytics')) {
    return request.abort();
  }

  // Stub a backend endpoint with deterministic data
  if (url.endsWith('/api/feature-flags')) {
    return request.respond({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ newCheckout: true, darkMode: false }),
    });
  }

  return request.continue();
});

await page.goto('https://app.example.com/dashboard', { waitUntil: 'networkidle2' });

Waiting on Responses and Extracting Data

// Wait for the specific XHR the page fires, then read its JSON
const [response] = await Promise.all([
  page.waitForResponse(
    (res) => res.url().includes('/api/search') && res.status() === 200,
  ),
  page.locator('input[name="q"]').fill('playwright'),
]);
const results = (await response.json()) as { items: { title: string }[] };

// Extract structured data from the DOM in one evaluate call
const rows = await page.$$eval('table#skills tbody tr', (trs) =>
  trs.map((tr) => ({
    name: tr.querySelector('td:nth-child(1)')?.textContent?.trim() ?? '',
    installs: Number(tr.querySelector('td:nth-child(2)')?.textContent ?? 0),
  })),
);
console.log(rows.filter((r) => r.installs > 100));

Reusable Page Helper for Flaky-Free Typing

import type { Page } from 'puppeteer';

export async function clearAndType(page: Page, selector: string, value: string): Promise<void> {
  const input = await page.waitForSelector(selector, { visible: true });
  await input!.click({ clickCount: 3 }); // select existing text
  await input!.press('Backspace');
  await input!.type(value, { delay: 20 });
}

Best Practices

• Pin the Puppeteer version; each release bundles a specific Chrome. Mismatched puppeteer-core + system Chrome is the top source of "works on my machine".
• Set page.setDefaultTimeout() once per page instead of passing { timeout } everywhere.
• In Docker, use the official ghcr.io/puppeteer/puppeteer image or install the documented dependency list — a bare node:20-slim will fail with cryptic shared-library errors.
• Reuse one Browser across many pages; launching Chrome costs 1-2 seconds, browser.newPage() costs milliseconds.
• Capture a screenshot in your catch block before rethrowing — page.screenshot({ path: 'failure.png' }) turns a CI mystery into a one-look diagnosis.
• For E2E test suites with assertions, fixtures, and retries, prefer Playwright; keep Puppeteer for scraping, PDF/screenshot services, and Chrome-extension automation where it excels.

Anti-Patterns

• page.waitForTimeout(3000) / sleep-based waits. Replace with waitForSelector, waitForResponse, or waitForFunction.
• headless: false committed to CI scripts. Headful Chrome needs a display server; CI dies with "Missing X server". Gate it behind an env var for local debugging only.
• Scraping inside page.evaluate with variables captured from Node scope. The callback serializes to the browser; closures over Node objects throw. Pass data as arguments: page.evaluate((sel) => ..., selector).
• One giant try/catch around the whole script with no finally close. Zombie Chrome processes exhaust CI memory.
• Enabling request interception and forgetting request.continue() in the default branch — every request hangs and the page never loads.
• Selectors built from generated class names like .css-1q2w3e. Use IDs, data-testid, ARIA roles, or stable attribute selectors.

When to Trigger This Skill

• The repo depends on puppeteer or puppeteer-core, or has scripts importing them.
• The user asks to scrape a website, generate PDFs from HTML, or capture screenshots programmatically.
• A headless-Chrome task in Docker/CI is failing with sandbox, /dev/shm, or missing-library errors.
• Automating Chrome-specific surfaces: extensions, DevTools protocol features, performance traces.
• Existing Puppeteer code is flaky and needs waits, interception, or lifecycle fixes.