Visual Regression Test Setup

Visual Regression Test Setup

Sets up automated visual regression testing to catch unintended UI changes through pixel-perfect screenshot comparison.

When to Use

• "Setup visual regression testing"
• "Add screenshot tests"
• "Prevent visual bugs"
• "Setup Percy/Chromatic"
• "Test UI changes automatically"

Instructions

1. Choose Testing Tool

Popular Options:

• Percy (BrowserStack) - Easy setup, CI/CD integration
• Chromatic (Storybook) - Best for component libraries
• Playwright - Free, self-hosted
• BackstopJS - Free, self-hosted
• Applitools - AI-powered

2. Setup Percy (Recommended for Most Projects)

Install:

npm install --save-dev @percy/cli @percy/playwright
# or for other frameworks
npm install --save-dev @percy/cypress
npm install --save-dev @percy/puppeteer
npm install --save-dev @percy/storybook

Playwright + Percy:

// tests/visual/homepage.spec.ts
import { test } from '@playwright/test';
import percySnapshot from '@percy/playwright';

test.describe('Homepage Visual Tests', () => {
  test('homepage desktop', async ({ page }) => {
    await page.goto('http://localhost:3000');
    await percySnapshot(page, 'Homepage - Desktop');
  });

  test('homepage mobile', async ({ page }) => {
    await page.setViewportSize({ width: 375, height: 667 });
    await page.goto('http://localhost:3000');
    await percySnapshot(page, 'Homepage - Mobile');
  });

  test('homepage tablet', async ({ page }) => {
    await page.setViewportSize({ width: 768, height: 1024 });
    await page.goto('http://localhost:3000');
    await percySnapshot(page, 'Homepage - Tablet');
  });

  test('dark mode', async ({ page }) => {
    await page.goto('http://localhost:3000');
    await page.click('[data-testid="theme-toggle"]');
    await page.waitForTimeout(500); // Wait for transition
    await percySnapshot(page, 'Homepage - Dark Mode');
  });
});

package.json:

{
  "scripts": {
    "test:visual": "percy exec -- playwright test tests/visual",
    "test:visual:update": "percy exec -- playwright test tests/visual --update-snapshots"
  }
}

.percy.yml:

version: 2
snapshot:
  widths:
    - 375   # Mobile
    - 768   # Tablet
    - 1280  # Desktop
  min-height: 1024
  percy-css: |
    /* Hide dynamic content */
    .timestamp,
    .random-number,
    [data-testid="current-time"] {
      visibility: hidden;
    }

GitHub Actions:

name: Visual Tests

on: [push, pull_request]

jobs:
  visual-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build

      - name: Run visual tests
        run: npm run test:visual
        env:
          PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}

3. Setup Chromatic (Best for Storybook)

Install:

npm install --save-dev chromatic

Setup Storybook (if not already):

npx storybook init

Run Chromatic:

npx chromatic --project-token=<your-project-token>

package.json:

{
  "scripts": {
    "chromatic": "chromatic --exit-zero-on-changes"
  }
}

GitHub Actions:

name: Chromatic

on: push

jobs:
  chromatic:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Required for Chromatic

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run Chromatic
        uses: chromaui/action@v1
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          exitZeroOnChanges: true  # Don't fail on visual changes
          autoAcceptChanges: main  # Auto-accept on main branch

Storybook Story Example:

// Button.stories.tsx
import type { Meta, StoryObj } from '@storybook/react';
import { Button } from './Button';

const meta: Meta<typeof Button> = {
  title: 'Components/Button',
  component: Button,
  parameters: {
    chromatic: {
      // Delay for animations
      delay: 300,
      // Test specific viewports
      viewports: [320, 768, 1200],
      // Disable animations
      disableSnapshot: false,
    },
  },
};

export default meta;
type Story = StoryObj<typeof Button>;

export const Primary: Story = {
  args: {
    variant: 'primary',
    children: 'Click me',
  },
};

export const AllStates: Story = {
  render: () => (
    <div style={{ display: 'flex', gap: '1rem', flexDirection: 'column' }}>
      <Button variant="primary">Normal</Button>
      <Button variant="primary" className="hover">Hover</Button>
      <Button variant="primary" disabled>Disabled</Button>
      <Button variant="primary" loading>Loading</Button>
    </div>
  ),
};

4. Setup Playwright Visual Comparisons (Free)

playwright.config.ts:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  snapshotDir: './tests/snapshots',

  expect: {
    toHaveScreenshot: {
      maxDiffPixels: 100,
      threshold: 0.2,
    },
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'mobile',
      use: { ...devices['iPhone 12'] },
    },
  ],

  webServer: {
    command: 'npm run dev',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
});

Test:

// tests/visual/homepage.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Visual Regression Tests', () => {
  test('homepage looks correct', async ({ page }) => {
    await page.goto('http://localhost:3000');

    // Wait for page to be fully loaded
    await page.waitForLoadState('networkidle');

    // Hide dynamic content
    await page.addStyleTag({
      content: '.timestamp { visibility: hidden; }'
    });

    // Take screenshot
    await expect(page).toHaveScreenshot('homepage.png');
  });

  test('button states', async ({ page }) => {
    await page.goto('http://localhost:3000/components');

    const button = page.locator('[data-testid="primary-button"]');

    // Normal state
    await expect(button).toHaveScreenshot('button-normal.png');

    // Hover state
    await button.hover();
    await expect(button).toHaveScreenshot('button-hover.png');

    // Focus state
    await button.focus();
    await expect(button).toHaveScreenshot('button-focus.png');
  });

  test('responsive design', async ({ page }) => {
    await page.goto('http://localhost:3000');

    // Desktop
    await page.setViewportSize({ width: 1920, height: 1080 });
    await expect(page).toHaveScreenshot('homepage-desktop.png');

    // Tablet
    await page.setViewportSize({ width: 768, height: 1024 });
    await expect(page).toHaveScreenshot('homepage-tablet.png');

    // Mobile
    await page.setViewportSize({ width: 375, height: 667 });
    await expect(page).toHaveScreenshot('homepage-mobile.png');
  });
});

Update snapshots:

npx playwright test --update-snapshots

5. Setup BackstopJS (Free, Self-Hosted)

Install:

npm install --save-dev backstopjs

Initialize:

npx backstop init

backstop.json:

{
  "id": "myapp_visual_tests",
  "viewports": [
    {
      "label": "phone",
      "width": 375,
      "height": 667
    },
    {
      "label": "tablet",
      "width": 768,
      "height": 1024
    },
    {
      "label": "desktop",
      "width": 1920,
      "height": 1080
    }
  ],
  "scenarios": [
    {
      "label": "Homepage",
      "url": "http://localhost:3000",
      "delay": 500,
      "misMatchThreshold": 0.1,
      "requireSameDimensions": true
    },
    {
      "label": "About Page",
      "url": "http://localhost:3000/about",
      "delay": 500
    },
    {
      "label": "Dark Mode",
      "url": "http://localhost:3000",
      "clickSelector": "[data-testid='theme-toggle']",
      "postInteractionWait": 1000,
      "delay": 500
    },
    {
      "label": "Form States",
      "url": "http://localhost:3000/contact",
      "selectors": ["form"],
      "hoverSelector": "button[type='submit']",
      "delay": 200
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "engine": "puppeteer",
  "report": ["browser", "CI"],
  "debug": false,
  "debugWindow": false
}

package.json scripts:

{
  "scripts": {
    "backstop:reference": "backstop reference",
    "backstop:test": "backstop test",
    "backstop:approve": "backstop approve"
  }
}

Usage:

# Create baseline screenshots
npm run backstop:reference

# Run tests (compare against baseline)
npm run backstop:test

# Approve changes (update baseline)
npm run backstop:approve

6. Best Practices

Handling Dynamic Content:

// Hide timestamps, random IDs, etc.
await page.addStyleTag({
  content: `
    .timestamp,
    .random-id,
    [data-dynamic="true"] {
      visibility: hidden !important;
    }
  `
});

// Or use Percy CSS
// .percy.yml
snapshot:
  percy-css: |
    .timestamp { display: none; }

Testing Animations:

// Disable animations for consistent screenshots
await page.addStyleTag({
  content: `
    *,
    *::before,
    *::after {
      animation-duration: 0s !important;
      animation-delay: 0s !important;
      transition-duration: 0s !important;
      transition-delay: 0s !important;
    }
  `
});

Testing Component States:

test('button all states', async ({ page }) => {
  await page.goto('http://localhost:3000/components');

  // Create a grid of all states
  await page.evaluate(() => {
    const container = document.querySelector('[data-testid="button-container"]');
    container.innerHTML = `
      <div style="display: grid; grid-template-columns: repeat(3, 1fr); gap: 1rem;">
        <button class="primary">Normal</button>
        <button class="primary hover">Hover</button>
        <button class="primary focus">Focus</button>
        <button class="primary active">Active</button>
        <button class="primary" disabled>Disabled</button>
        <button class="primary loading">Loading</button>
      </div>
    `;
  });

  await expect(page.locator('[data-testid="button-container"]'))
    .toHaveScreenshot('button-all-states.png');
});

7. CI/CD Integration Tips

Parallel Testing:

# GitHub Actions
jobs:
  visual-tests:
    strategy:
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - run: npx playwright test --project=${{ matrix.browser }}

Only Run on UI Changes:

on:
  pull_request:
    paths:
      - 'src/components/**'
      - 'src/pages/**'
      - 'src/styles/**'

Automatic Approval on Main:

- name: Auto-approve on main
  if: github.ref == 'refs/heads/main'
  run: npm run backstop:approve

8. Maintenance

Review Process:

# Visual Regression Review Checklist

When visual tests fail:

1. [ ] Review the diff in Percy/Chromatic dashboard
2. [ ] Verify changes are intentional
3. [ ] Check all viewports (mobile, tablet, desktop)
4. [ ] Test in different browsers if applicable
5. [ ] Approve changes or fix issues
6. [ ] Update baseline snapshots

If changes are intentional:
- Approve in Percy/Chromatic dashboard
- Or run `npm run test:visual:update`

If changes are bugs:
- Fix the CSS/component
- Re-run tests to verify fix

Best Practices

DO:

• Test multiple viewports
• Hide dynamic content
• Disable animations
• Test critical user flows
• Review diffs carefully
• Update baselines deliberately
• Test dark mode
• Test component states

DON'T:

• Test every page
• Ignore flaky tests
• Auto-approve everything
• Skip mobile testing
• Test with live data
• Forget to wait for loading
• Screenshot entire pages only
• Ignore small differences

Checklist

• Visual testing tool selected
• Dependencies installed
• Configuration created
• Baseline snapshots created
• CI/CD integration added
• Dynamic content handled
• Multiple viewports tested
• Review process documented
• Team trained on workflow