Puppeteer Browser Automation
This skill makes an AI agent write reliable Puppeteer scripts: launching Chrome with the right flags, navigating and interacting without race conditions, capturing screenshots and PDFs, and intercepting network requests to mock or block traffic. Trigger it when a project uses puppeteer or puppeteer-core, or when the user asks to scrape a page, generate a PDF from HTML, screenshot a site, or automate Chrome without a full test framework.
Core Principles
- Every action must wait for its precondition.
page.click() immediately after page.goto() races against rendering. Use page.locator() (auto-waiting, Puppeteer 21+) or explicit waitForSelector before every interaction.
- Never use fixed sleeps.
await new Promise(r => setTimeout(r, 3000)) is either too short (flaky) or too long (slow). Wait on selectors, network idle, or response predicates instead.
- Always close the browser in
finally. A script that throws before browser.close() leaks a Chrome process. In CI those zombies accumulate until the runner dies.
- Set
waitUntil deliberately. load waits for every image and font; domcontentloaded is enough for interaction; networkidle2 is for SPAs that fetch after load. Pick per page, do not cargo-cult networkidle0.
- Combine navigation-triggering actions with
Promise.all. Clicking a link then awaiting waitForNavigation separately misses fast navigations. Start the wait before the click.
- Request interception is your mock layer. Block analytics and images for speed, stub API responses for determinism — no proxy server needed.
Setup
npm install --save-dev puppeteer typescript tsx
// src/browser.ts
import puppeteer, { Browser } from 'puppeteer';
export async function launchBrowser(): Promise<Browser> {
return puppeteer.launch({
headless: true,
args: [
'--no-sandbox', // required in most Docker/CI containers
'--disable-dev-shm-usage', // /dev/shm is 64MB in Docker; avoids renderer crashes
'--disable-gpu',
'--window-size=1366,768',
],
defaultViewport: { width: 1366, height: 768 },
});
}
A complete script with correct lifecycle handling:
// src/check-login.ts
import { launchBrowser } from './browser';
async function main(): Promise<void> {
const browser = await launchBrowser();
try {
const page = await browser.newPage();
page.setDefaultTimeout(15_000);
await page.goto('https://practice.expandtesting.com/login', {
waitUntil: 'domcontentloaded',
});
// locator() auto-waits for visibility and stability before acting
await page.locator('#username').fill('practice');
await page.locator('#password').fill('SuperSecretPassword!');
await Promise.all([
page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
page.locator('button[type="submit"]').click(),
]);
const flash = await page.locator('#flash').waitHandle();
const text = await flash.evaluate((el) => el.textContent?.trim());
if (!text?.includes('You logged into a secure area')) {
throw new Error(`Login failed, flash message: ${text}`);
}
console.log('Login OK');
} finally {
await browser.close();
}
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
npx tsx src/check-login.ts
Patterns
Screenshots and PDF Generation
import { launchBrowser } from './browser';
const browser = await launchBrowser();
try {
const page = await browser.newPage();
await page.goto('https://qaskills.sh', { waitUntil: 'networkidle2' });
// Full-page screenshot
await page.screenshot({ path: 'homepage.png', fullPage: true });
// Screenshot of one element only
const hero = await page.waitForSelector('main section:first-of-type');
await hero!.screenshot({ path: 'hero.png' });
// PDF requires headless mode; emulate print CSS first
await page.emulateMediaType('print');
await page.pdf({
path: 'homepage.pdf',
format: 'A4',
printBackground: true,
margin: { top: '20mm', bottom: '20mm', left: '15mm', right: '15mm' },
});
} finally {
await browser.close();
}
Request Interception: Block Noise, Stub APIs
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (request) => {
const url = request.url();
const type = request.resourceType();
// Block images, fonts, and trackers for a 3-5x speedup on content scraping
if (type === 'image' || type === 'font' || url.includes('google-analytics')) {
return request.abort();
}
// Stub a backend endpoint with deterministic data
if (url.endsWith('/api/feature-flags')) {
return request.respond({
status: 200,
contentType: 'application/json',
body: JSON.stringify({ newCheckout: true, darkMode: false }),
});
}
return request.continue();
});
await page.goto('https://app.example.com/dashboard', { waitUntil: 'networkidle2' });
Waiting on Responses and Extracting Data
// Wait for the specific XHR the page fires, then read its JSON
const [response] = await Promise.all([
page.waitForResponse(
(res) => res.url().includes('/api/search') && res.status() === 200,
),
page.locator('input[name="q"]').fill('playwright'),
]);
const results = (await response.json()) as { items: { title: string }[] };
// Extract structured data from the DOM in one evaluate call
const rows = await page.$$eval('table#skills tbody tr', (trs) =>
trs.map((tr) => ({
name: tr.querySelector('td:nth-child(1)')?.textContent?.trim() ?? '',
installs: Number(tr.querySelector('td:nth-child(2)')?.textContent ?? 0),
})),
);
console.log(rows.filter((r) => r.installs > 100));
Reusable Page Helper for Flaky-Free Typing
import type { Page } from 'puppeteer';
export async function clearAndType(page: Page, selector: string, value: string): Promise<void> {
const input = await page.waitForSelector(selector, { visible: true });
await input!.click({ clickCount: 3 }); // select existing text
await input!.press('Backspace');
await input!.type(value, { delay: 20 });
}
Best Practices
- Pin the Puppeteer version; each release bundles a specific Chrome. Mismatched
puppeteer-core + system Chrome is the top source of "works on my machine".
- Set
page.setDefaultTimeout() once per page instead of passing { timeout } everywhere.
- In Docker, use the official
ghcr.io/puppeteer/puppeteer image or install the documented dependency list — a bare node:20-slim will fail with cryptic shared-library errors.
- Reuse one
Browser across many pages; launching Chrome costs 1-2 seconds, browser.newPage() costs milliseconds.
- Capture a screenshot in your
catch block before rethrowing — page.screenshot({ path: 'failure.png' }) turns a CI mystery into a one-look diagnosis.
- For E2E test suites with assertions, fixtures, and retries, prefer Playwright; keep Puppeteer for scraping, PDF/screenshot services, and Chrome-extension automation where it excels.
Anti-Patterns
page.waitForTimeout(3000) / sleep-based waits. Replace with waitForSelector, waitForResponse, or waitForFunction.
headless: false committed to CI scripts. Headful Chrome needs a display server; CI dies with "Missing X server". Gate it behind an env var for local debugging only.
- Scraping inside
page.evaluate with variables captured from Node scope. The callback serializes to the browser; closures over Node objects throw. Pass data as arguments: page.evaluate((sel) => ..., selector).
- One giant try/catch around the whole script with no
finally close. Zombie Chrome processes exhaust CI memory.
- Enabling request interception and forgetting
request.continue() in the default branch — every request hangs and the page never loads.
- Selectors built from generated class names like
.css-1q2w3e. Use IDs, data-testid, ARIA roles, or stable attribute selectors.
When to Trigger This Skill
- The repo depends on
puppeteer or puppeteer-core, or has scripts importing them.
- The user asks to scrape a website, generate PDFs from HTML, or capture screenshots programmatically.
- A headless-Chrome task in Docker/CI is failing with sandbox,
/dev/shm, or missing-library errors.
- Automating Chrome-specific surfaces: extensions, DevTools protocol features, performance traces.
- Existing Puppeteer code is flaky and needs waits, interception, or lifecycle fixes.
1---2name: puppeteer-browser-automation3description: Automate Chrome with Puppeteer for scraping, screenshots, PDF generation, and E2E checks — correct launch options, reliable waiting with locators and waitForSelector, request interception, and headless CI execution.4version: 1.0.05author: thetestingacademy6license: MIT7tags: [puppeteer, browser-automation, headless-chrome, scraping, screenshots, pdf, e2e, devtools, ci]8testingTypes: [e2e]9frameworks: [puppeteer, jest]10languages: [typescript, javascript]11domains: [web]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]13metadata:14 source: https://skillmd.com/skills/pramoddutta/puppeteer-browser-automation15---1617# Puppeteer Browser Automation1819This skill makes an AI agent write reliable Puppeteer scripts: launching Chrome with the right flags, navigating and interacting without race conditions, capturing screenshots and PDFs, and intercepting network requests to mock or block traffic. Trigger it when a project uses `puppeteer` or `puppeteer-core`, or when the user asks to scrape a page, generate a PDF from HTML, screenshot a site, or automate Chrome without a full test framework.2021## Core Principles22231. **Every action must wait for its precondition.** `page.click()` immediately after `page.goto()` races against rendering. Use `page.locator()` (auto-waiting, Puppeteer 21+) or explicit `waitForSelector` before every interaction.242. **Never use fixed sleeps.** `await new Promise(r => setTimeout(r, 3000))` is either too short (flaky) or too long (slow). Wait on selectors, network idle, or response predicates instead.253. **Always close the browser in `finally`.** A script that throws before `browser.close()` leaks a Chrome process. In CI those zombies accumulate until the runner dies.264. **Set `waitUntil` deliberately.** `load` waits for every image and font; `domcontentloaded` is enough for interaction; `networkidle2` is for SPAs that fetch after load. Pick per page, do not cargo-cult `networkidle0`.275. **Combine navigation-triggering actions with `Promise.all`.** Clicking a link then awaiting `waitForNavigation` separately misses fast navigations. Start the wait before the click.286. **Request interception is your mock layer.** Block analytics and images for speed, stub API responses for determinism — no proxy server needed.2930## Setup3132```bash33npm install --save-dev puppeteer typescript tsx34```3536```typescript37// src/browser.ts38import puppeteer, { Browser } from 'puppeteer';3940export async function launchBrowser(): Promise<Browser> {41 return puppeteer.launch({42 headless: true,43 args: [44 '--no-sandbox', // required in most Docker/CI containers45 '--disable-dev-shm-usage', // /dev/shm is 64MB in Docker; avoids renderer crashes46 '--disable-gpu',47 '--window-size=1366,768',48 ],49 defaultViewport: { width: 1366, height: 768 },50 });51}52```5354A complete script with correct lifecycle handling:5556```typescript57// src/check-login.ts58import { launchBrowser } from './browser';5960async function main(): Promise<void> {61 const browser = await launchBrowser();62 try {63 const page = await browser.newPage();64 page.setDefaultTimeout(15_000);6566 await page.goto('https://practice.expandtesting.com/login', {67 waitUntil: 'domcontentloaded',68 });6970 // locator() auto-waits for visibility and stability before acting71 await page.locator('#username').fill('practice');72 await page.locator('#password').fill('SuperSecretPassword!');7374 await Promise.all([75 page.waitForNavigation({ waitUntil: 'domcontentloaded' }),76 page.locator('button[type="submit"]').click(),77 ]);7879 const flash = await page.locator('#flash').waitHandle();80 const text = await flash.evaluate((el) => el.textContent?.trim());81 if (!text?.includes('You logged into a secure area')) {82 throw new Error(`Login failed, flash message: ${text}`);83 }84 console.log('Login OK');85 } finally {86 await browser.close();87 }88}8990main().catch((err) => {91 console.error(err);92 process.exit(1);93});94```9596```bash97npx tsx src/check-login.ts98```99100## Patterns101102### Screenshots and PDF Generation103104```typescript105import { launchBrowser } from './browser';106107const browser = await launchBrowser();108try {109 const page = await browser.newPage();110 await page.goto('https://qaskills.sh', { waitUntil: 'networkidle2' });111112 // Full-page screenshot113 await page.screenshot({ path: 'homepage.png', fullPage: true });114115 // Screenshot of one element only116 const hero = await page.waitForSelector('main section:first-of-type');117 await hero!.screenshot({ path: 'hero.png' });118119 // PDF requires headless mode; emulate print CSS first120 await page.emulateMediaType('print');121 await page.pdf({122 path: 'homepage.pdf',123 format: 'A4',124 printBackground: true,125 margin: { top: '20mm', bottom: '20mm', left: '15mm', right: '15mm' },126 });127} finally {128 await browser.close();129}130```131132### Request Interception: Block Noise, Stub APIs133134```typescript135const page = await browser.newPage();136await page.setRequestInterception(true);137138page.on('request', (request) => {139 const url = request.url();140 const type = request.resourceType();141142 // Block images, fonts, and trackers for a 3-5x speedup on content scraping143 if (type === 'image' || type === 'font' || url.includes('google-analytics')) {144 return request.abort();145 }146147 // Stub a backend endpoint with deterministic data148 if (url.endsWith('/api/feature-flags')) {149 return request.respond({150 status: 200,151 contentType: 'application/json',152 body: JSON.stringify({ newCheckout: true, darkMode: false }),153 });154 }155156 return request.continue();157});158159await page.goto('https://app.example.com/dashboard', { waitUntil: 'networkidle2' });160```161162### Waiting on Responses and Extracting Data163164```typescript165// Wait for the specific XHR the page fires, then read its JSON166const [response] = await Promise.all([167 page.waitForResponse(168 (res) => res.url().includes('/api/search') && res.status() === 200,169 ),170 page.locator('input[name="q"]').fill('playwright'),171]);172const results = (await response.json()) as { items: { title: string }[] };173174// Extract structured data from the DOM in one evaluate call175const rows = await page.$$eval('table#skills tbody tr', (trs) =>176 trs.map((tr) => ({177 name: tr.querySelector('td:nth-child(1)')?.textContent?.trim() ?? '',178 installs: Number(tr.querySelector('td:nth-child(2)')?.textContent ?? 0),179 })),180);181console.log(rows.filter((r) => r.installs > 100));182```183184### Reusable Page Helper for Flaky-Free Typing185186```typescript187import type { Page } from 'puppeteer';188189export async function clearAndType(page: Page, selector: string, value: string): Promise<void> {190 const input = await page.waitForSelector(selector, { visible: true });191 await input!.click({ clickCount: 3 }); // select existing text192 await input!.press('Backspace');193 await input!.type(value, { delay: 20 });194}195```196197## Best Practices198199- Pin the Puppeteer version; each release bundles a specific Chrome. Mismatched `puppeteer-core` + system Chrome is the top source of "works on my machine".200- Set `page.setDefaultTimeout()` once per page instead of passing `{ timeout }` everywhere.201- In Docker, use the official `ghcr.io/puppeteer/puppeteer` image or install the documented dependency list — a bare `node:20-slim` will fail with cryptic shared-library errors.202- Reuse one `Browser` across many pages; launching Chrome costs 1-2 seconds, `browser.newPage()` costs milliseconds.203- Capture a screenshot in your `catch` block before rethrowing — `page.screenshot({ path: 'failure.png' })` turns a CI mystery into a one-look diagnosis.204- For E2E test suites with assertions, fixtures, and retries, prefer Playwright; keep Puppeteer for scraping, PDF/screenshot services, and Chrome-extension automation where it excels.205206## Anti-Patterns207208- **`page.waitForTimeout(3000)` / sleep-based waits.** Replace with `waitForSelector`, `waitForResponse`, or `waitForFunction`.209- **`headless: false` committed to CI scripts.** Headful Chrome needs a display server; CI dies with "Missing X server". Gate it behind an env var for local debugging only.210- **Scraping inside `page.evaluate` with variables captured from Node scope.** The callback serializes to the browser; closures over Node objects throw. Pass data as arguments: `page.evaluate((sel) => ..., selector)`.211- **One giant try/catch around the whole script with no `finally` close.** Zombie Chrome processes exhaust CI memory.212- **Enabling request interception and forgetting `request.continue()`** in the default branch — every request hangs and the page never loads.213- **Selectors built from generated class names** like `.css-1q2w3e`. Use IDs, `data-testid`, ARIA roles, or stable attribute selectors.214215## When to Trigger This Skill216217- The repo depends on `puppeteer` or `puppeteer-core`, or has scripts importing them.218- The user asks to scrape a website, generate PDFs from HTML, or capture screenshots programmatically.219- A headless-Chrome task in Docker/CI is failing with sandbox, `/dev/shm`, or missing-library errors.220- Automating Chrome-specific surfaces: extensions, DevTools protocol features, performance traces.221- Existing Puppeteer code is flaky and needs waits, interception, or lifecycle fixes.