Test MCP Server

Test MCP Server

sunpeak includes a testing framework that works with any MCP server, regardless of language or framework. It provides four testing layers: e2e tests (inspector-based), visual regression, live tests (against real ChatGPT), and evals (multi-model tool calling).

For sunpeak app projects, testing integrates automatically. For non-sunpeak MCP servers (Python, Go, Rust, etc.), sunpeak test init scaffolds a self-contained test directory.

Getting Started

sunpeak test init                        # Interactive setup (detects project type)
sunpeak test init --server http://localhost:8000/mcp  # URL-based server
sunpeak test init --server "python server.py"         # Command-based server

sunpeak test init detects three project types:

• sunpeak projects -- Adds defineConfig() and eval boilerplate
• JS/TS projects -- Adds Playwright config, smoke test, and evals at project root
• Non-JS projects -- Creates a self-contained tests/sunpeak/ directory with its own package.json

Getting Reference Code

Clone the sunpeak repo for working test examples:

git clone --depth 1 https://github.com/Alignbase/sunpeak /tmp/sunpeak

Test examples live at /tmp/sunpeak/packages/sunpeak/template/tests/. This includes e2e tests, simulations, evals, and live tests.

Commands

sunpeak inspect              # Inspect any MCP server in the inspector (standalone)
sunpeak test                 # Run unit (if configured) + e2e tests
sunpeak test --e2e           # Run e2e tests only (Playwright)
sunpeak test --visual        # Run e2e tests with visual regression comparison
sunpeak test --visual --update  # Update visual regression baselines
sunpeak test init            # Scaffold test infrastructure into a project
sunpeak test --live          # Run live tests against real ChatGPT (requires tunnel + browser session)
sunpeak test --eval          # Run evals against multiple LLM models (requires API keys)
sunpeak test --unit          # Run unit tests (sunpeak app framework only, not standalone)

Flags are additive: --e2e --live --eval runs all three. --update implies --visual. --eval and --live are never included in the default run (they cost money). --unit is only for sunpeak app framework projects (created with sunpeak new), not standalone testing.

E2E Tests with the mcp and inspector Fixtures

Import test and expect from sunpeak/test. The mcp fixture provides protocol-level methods, and the inspector fixture handles rendering, double-iframe traversal, URL construction, and host selection. Tests run automatically across ChatGPT and Claude hosts via Playwright projects.

import { test, expect } from 'sunpeak/test';

test('renders weather card', async ({ inspector }) => {
  const result = await inspector.renderTool('show-weather');
  const app = result.app();
  await expect(app.locator('h1')).toHaveText('Austin');
});

test('renders in dark mode', async ({ inspector }) => {
  const result = await inspector.renderTool('show-weather', {}, { theme: 'dark' });
  const app = result.app();
  await expect(app.locator('h1')).toBeVisible();
});

test('loads without console errors', async ({ inspector }) => {
  const errors: string[] = [];
  inspector.page.on('console', (msg) => {
    if (msg.type() === 'error') errors.push(msg.text());
  });

  const result = await inspector.renderTool('show-weather', {}, { theme: 'dark' });
  const app = result.app();
  await expect(app.locator('h1')).toBeVisible();

  const unexpectedErrors = errors.filter(
    (e) =>
      !e.includes('[IframeResource]') &&
      !e.includes('mcp') &&
      !e.includes('PostMessage') &&
      !e.includes('connect')
  );
  expect(unexpectedErrors).toHaveLength(0);
});

test('pip mode (skip on Claude)', async ({ inspector }) => {
  test.skip(inspector.host === 'claude', 'Claude does not support PiP');
  const result = await inspector.renderTool('show-weather', {}, { displayMode: 'pip' });
  await expect(result.app().locator('h1')).toBeVisible({ timeout: 5000 });
});

mcp Fixture API

Protocol-level methods (raw MCP data, no rendering):

inspector Fixture API

Rendering methods and properties:

InspectorResult API

MCP-Native Matchers

renderTool

inspector.renderTool renders the tool result in the inspector and returns an InspectorResult. With input, the tool is called on the real server with those arguments. Without input, simulation fixture data is used when available, or the real server is called with empty args.

// Calls the real server with arguments and renders
const result = await inspector.renderTool('search', { query: 'headphones' });

// Uses simulation fixture data, or calls server with empty args
const result = await inspector.renderTool('show-albums');

Configuring Timeouts

Set default timeouts in Playwright config for servers that need more time:

export default defineConfig({
  server: { command: 'python', args: ['server.py'] },
  timeout: 120_000,  // Server startup timeout (default: 60s)
  use: {
    mcpTimeout: 30_000,    // renderTool iframe timeout (default: 15s)
  },
});

Visual Regression Testing

Use result.screenshot() to capture and compare screenshots against saved baselines. Comparisons only run with sunpeak test --visual. Without it, screenshot() silently skips, so you can include it in regular e2e tests.

By default, result.screenshot() captures the rendered app inside the double-iframe, not the inspector chrome. Keep visual baselines scoped to the app unless the test is explicitly about the inspector UI.

import { test, expect } from 'sunpeak/test';

test('albums renders correctly', async ({ inspector }) => {
  const result = await inspector.renderTool('show-albums', {}, { theme: 'light' });
  const app = result.app();
  await expect(app.locator('button:has-text("Summer Slice")')).toBeVisible();

  await result.screenshot('albums-light');
});

screenshot() options:

All Playwright toHaveScreenshot options are passed through.

Configure project-wide visual defaults:

import { defineConfig } from 'sunpeak/test/config';
export default defineConfig({
  visual: {
    threshold: 0.2,
    maxDiffPixelRatio: 0.05,
  },
});

Playwright Config

// playwright.config.ts
import { defineConfig } from 'sunpeak/test/config';
export default defineConfig();
// Creates per-host projects (chatgpt, claude). Tests run once per host automatically.

For non-sunpeak MCP servers, pass a server option:

import { defineConfig } from 'sunpeak/test/config';
export default defineConfig({
  server: {
    url: 'http://localhost:8000/mcp',
  },
});

// Or with a command (stdio server):
export default defineConfig({
  server: {
    command: 'python',
    args: ['server.py'],
    env: { API_KEY: 'test-key' },  // Extra environment variables
    cwd: './backend',               // Working directory
  },
  timeout: 120_000,  // Server startup timeout in ms (default: 60s)
});

Locator Rules

Resource content renders inside a double-iframe (outer sandbox proxy + inner app iframe). In e2e tests:

• Use result.app().locator(...) (from inspector.renderTool()) for resource content. This handles the double-iframe sandbox architecture.
• Use inspector.page.locator(...) only for inspector chrome elements (header, #root, sidebar controls).

Simulations

E2e tests consume simulation fixtures defined in tests/simulations/*.json. For sunpeak projects, simulations are part of the app project structure (see the create-sunpeak-app skill for the simulation file format). For non-sunpeak servers, renderTool connects to the live server via the configured server option.

Live Testing (against real ChatGPT)

Live tests validate MCP Apps inside real ChatGPT. They use Playwright to open the user's browser, send messages that trigger tool calls, and assert on the rendered app iframe.

Before the first run, make sure Developer mode is enabled. If needed, click the bottom-left user menu and select Settings > Security and login > Developer mode. Then return to the ChatGPT homepage, click Plugins in the sidebar, and open the app if it is already there or select + to create it with the tunneled /mcp URL. Set the app name to the package.json name because live tests use that name when invoking the app.

Live Test Pattern

One spec file per resource. Import test and expect from sunpeak/test/live -- the live fixture handles login, MCP refresh, and host-specific message formatting.

// tests/live/weather.spec.ts
import { test, expect } from 'sunpeak/test/live';

test('weather tool renders forecast', async ({ live }) => {
  const app = await live.invoke('show me the weather in Austin');
  await expect(app.locator('h1')).toBeVisible();
});

Config is a one-liner:

// tests/live/playwright.config.ts
import { defineLiveConfig } from 'sunpeak/test/live/config';
export default defineLiveConfig();
// Add hosts: defineLiveConfig({ hosts: ['chatgpt', 'claude'] })
// Generates one Playwright project per host. Tests switch themes internally via live.setColorScheme().

live Fixture API

Running

# Requires: tunnel running (ngrok http 8000) + logged into ChatGPT in your browser
pnpm test:live

# Or via validate pipeline
sunpeak validate --live

The browser opens visibly -- headless mode is blocked by chatgpt.com's bot detection.

The live test runner imports your browser session, starts sunpeak dev --prod-resources, and refreshes the MCP server connection from its ChatGPT plugin details once in globalSetup before all workers. Tests run in parallel -- each test gets its own chat window.

If auth fails: If tests report "Not logged into ChatGPT", delete .auth/ and re-run pnpm test:live -- a browser window will open for you to log in again.

Evals (Multi-Model Tool Calling)

Evals test whether different LLMs call your tools correctly. They connect to your MCP server, discover tools via MCP protocol, and send prompts to multiple models to check tool calling behavior. Each case can include App Context for follow-up turns that depend on model-visible UI state. Each case runs N times per model to measure reliability.

Setup

pnpm add ai @ai-sdk/openai @ai-sdk/anthropic @ai-sdk/google

Copy tests/evals/.env.example to tests/evals/.env and add your API keys. The .env file is gitignored and loaded automatically when running evals. For sunpeak projects, the dev server starts automatically.

Configuration (tests/evals/eval.config.ts)

import { defineEvalConfig } from 'sunpeak/eval';

// API keys are loaded automatically from tests/evals/.env (gitignored).

export default defineEvalConfig({
  // Server is auto-detected for sunpeak projects.
  // For non-sunpeak projects: server: 'http://localhost:8000/mcp',

  models: ['gpt-4o', 'gpt-4o-mini', 'o4-mini', 'claude-sonnet-4-20250514', 'gemini-2.0-flash'],
  defaults: {
    runs: 10,
    maxSteps: 1,
    temperature: 0,
    timeout: 30_000,
  },
});

Writing Evals (tests/evals/*.eval.ts)

import { expect } from 'vitest';
import { defineEval } from 'sunpeak/eval';

export default defineEval({
  cases: [
    {
      name: 'food category request',
      prompt: 'Show me photos from my Austin pizza tour',
      expect: {
        tool: 'show-albums',
        args: { search: expect.stringMatching(/pizza|austin/i) },
      },
    },
    {
      name: 'multi-step flow',
      prompt: 'Write a post for X and LinkedIn',
      maxSteps: 3,
      expect: [
        { tool: 'review-post' },
        { tool: 'publish-post' },
      ],
    },
    {
      name: 'follow-up uses selected app state',
      prompt: 'Book this one',
      appContext: {
        structuredContent: {
          selectedFlight: { carrier: 'delta', flightNumber: 'DL123' },
        },
      },
      expect: {
        tool: 'book-flight',
        args: { carrier: 'delta' },
      },
    },
    {
      name: 'custom assertion',
      prompt: 'Show me vacation photos',
      assert: (result) => {
        expect(result.toolCalls).toHaveLength(1);
        expect(result.toolCalls[0].name).toBe('show-albums');
      },
    },
  ],
});

Three assertion levels: single tool (expect: { tool, args }), ordered sequence (expect: [...]), or custom function (assert: (result) => { ... }). Args use partial matching -- extra keys in the actual call are allowed. Use appContext to seed structuredContent or content that the MCP App would normally share with the model through updateModelContext.

Running

sunpeak test --eval                          # All evals
sunpeak test --eval tests/evals/albums.eval.ts  # Single file

Not included in the default sunpeak test run (costs money, like --live).

Common Mistakes

1. Wrong Playwright locator -- Use result.app().locator(...) (from inspector.renderTool()) for resource content. This handles the double-iframe sandbox architecture. Use inspector.page.locator(...) only for inspector chrome elements.
2. Simulation tool mismatch -- The "tool" field in simulation JSON must match a tool filename in src/tools/ (e.g. "tool": "show-weather" matches src/tools/show-weather.ts).
3. Missing console error filter -- When testing for console errors, always filter out expected MCP handshake errors ([IframeResource], mcp, PostMessage, connect).

Export Paths

Migrating from older versions

When upgrading sunpeak, check for deprecated API patterns in test files and update them. This section lists breaking changes by version.

0.20.0: Fixture split (mcp + inspector)

The single mcp fixture was split into two: mcp (MCP protocol) and inspector (rendering).

How to identify tests that need migration:

• { mcp } destructuring where the test uses result.app(), .screenshot(), theme, displayMode, or prodResources → change to { inspector } and use inspector.renderTool
• { mcp } destructuring where the test only uses callTool without .app() or rendering options → keep as { mcp }, this is the protocol-level API (no change needed)
• Tests using mcp.callTool('name', {}, { theme: 'dark' }) → the empty {} second arg was ignored before; now change to inspector.renderTool('name', undefined, { theme: 'dark' })

New protocol methods added to mcp: listTools(), listResources(), readResource(uri).

New InspectorResult fields: source ('fixture' | 'server'), screenshot() method.

References

• sunpeak Documentation
• MCP Apps Documentation
• MCP Apps SDK