Agent Test

Write agent tests: $ARGUMENTS

Before Starting

1. Read existing test examples:
   • /Users/joshuashepherd/Desktop/dev/repos/ai-lab-agent/tests/ — unit tests for catalog, router, citations, tools, context
   • /Users/joshuashepherd/Desktop/dev/repos/ai-lab-agent/e2e/ — Playwright endpoint specs
   • src/lib/ai-lab/catalog.test.ts — catalog validation tests (local)
2. Read the smoke test patterns:
   • /Users/joshuashepherd/Desktop/dev/repos/ai-lab-agent/smoke-test-chat.ts — tool-triggering prompts
   • /Users/joshuashepherd/Desktop/dev/repos/ai-lab-agent/smoke-test-context.ts — context propagation
   • /Users/joshuashepherd/Desktop/dev/repos/ai-lab-agent/smoke-test-fidelity.ts — book fidelity (file_search + citations)
3. Check test infrastructure:
   • Run pnpm test:run for Vitest (unit tests in tests/unit/)
   • Run pnpm test:e2e for Playwright (e2e tests in tests/e2e/)

Test Types

1. Unit Tests (Vitest)

Location: tests/unit/agents/

Tool tests — verify tool execution and output shape:

import { describe, it, expect, vi } from 'vitest';
import { myTool } from '@/agents/my-agent/tools';

describe('myTool', () => {
  it('returns structured results for valid input', async () => {
    const result = await myTool.execute({ query: 'test query' });
    expect(result).toHaveProperty('data');
    expect(Array.isArray(result.data)).toBe(true);
  });

  it('returns empty results when service is unavailable', async () => {
    vi.stubEnv('EXTERNAL_API_URL', '');
    const result = await myTool.execute({ query: 'test' });
    expect(result.data).toEqual([]);
  });

  it('respects maxResults parameter', async () => {
    const result = await myTool.execute({ query: 'test', maxResults: 3 });
    expect(result.data.length).toBeLessThanOrEqual(3);
  });
});

Instruction tests — verify prompt composition:

import { describe, it, expect } from 'vitest';
import { generateInstructions } from '@/agents/my-agent/instructions';

describe('generateInstructions', () => {
  it('includes core identity section', () => {
    const instructions = generateInstructions({ theme: 'metanoia', mode: 'teacher', style: 'conversation' });
    expect(instructions).toContain('You are');
    expect(instructions).toContain('voice markers');
  });

  it('adapts to theme', () => {
    const metanoia = generateInstructions({ theme: 'metanoia', mode: 'teacher', style: 'conversation' });
    const movemental = generateInstructions({ theme: 'movemental', mode: 'teacher', style: 'conversation' });
    expect(metanoia).not.toEqual(movemental);
  });

  it('includes dynamic context when provided', () => {
    const instructions = generateInstructions(
      { theme: 'metanoia', mode: 'teacher', style: 'conversation' },
      { userName: 'Alice' }
    );
    expect(instructions).toContain('Alice');
  });
});

Router tests — verify intent classification:

import { describe, it, expect } from 'vitest';
import { routeRetrieval } from '@/agents/my-agent/retrieval/router';

describe('routeRetrieval', () => {
  it('classifies quote requests', () => {
    const result = routeRetrieval('What does Alan say about missional ecclesiology?', {});
    expect(result.intent).toBe('QUOTE_REQUEST');
  });

  it('classifies book-specific queries', () => {
    const result = routeRetrieval('Summarize chapter 3 of Forgotten Ways', { booksFocus: ['forgotten-ways'] });
    expect(result.intent).toBe('BOOK_SPECIFIC');
    expect(result.temperatureOverride).toBeLessThanOrEqual(0.3);
  });

  it('returns GENERAL for casual conversation', () => {
    const result = routeRetrieval('How are you today?', {});
    expect(result.intent).toBe('GENERAL_CONVERSATION');
  });
});

Citation tests — verify parsing and rendering:

describe('parseSource', () => {
  it('extracts book slug and chapter from file path', () => {
    const source = parseSource('books/forgotten-ways/ch03-apostolic-environment.md');
    expect(source.bookSlug).toBe('forgotten-ways');
    expect(source.chapter).toBe(3);
  });
});

Guardrail tests — verify pass/reject behavior:

describe('contentFilter', () => {
  it('blocks script tags', async () => {
    const guardrail = contentFilter();
    const result = await guardrail.validate('<script>alert("xss")</script>');
    expect(result.passed).toBe(false);
  });

  it('passes normal text', async () => {
    const guardrail = contentFilter();
    const result = await guardrail.validate('Tell me about missional leadership');
    expect(result.passed).toBe(true);
  });
});

2. Smoke Tests (CLI scripts)

Location: scripts/smoke-tests/ or project root

Smoke tests hit the running agent endpoint and assert behavior:

// smoke-test-tool-trigger.ts
const response = await fetch(`${BASE_URL}/api/agents/chat`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${API_KEY}` },
  body: JSON.stringify({
    message: 'What does Alan Hirsch write about APEST in The Permanent Revolution?',
    context: { booksFocus: ['permanent-revolution'] },
  }),
});

const events = await parseSSEStream(response);
const toolCalls = events.filter(e => e.type === 'tool_call');
const doneEvent = events.find(e => e.type === 'done');

assert(toolCalls.length > 0, 'Expected at least one tool call for book query');
assert(toolCalls.some(t => t.name === 'file_search' || t.name === 'search_books'), 'Expected retrieval tool');
assert(doneEvent?.finalOutput?.includes('Source'), 'Expected citations in output');

3. E2E Tests (Playwright)

Location: tests/e2e/

import { test, expect } from '@playwright/test';

test('chat endpoint returns valid SSE stream', async ({ request }) => {
  const response = await request.post('/api/agents/chat', {
    data: { message: 'Hello' },
    headers: { Authorization: `Bearer ${process.env.API_KEY}` },
  });
  expect(response.status()).toBe(200);
  expect(response.headers()['content-type']).toContain('text/event-stream');

  const body = await response.text();
  expect(body).toContain('"type":"text_delta"');
  expect(body).toContain('"type":"done"');
});

Running Tests

# Unit tests
pnpm test:run -- agents           # Run all agent tests
pnpm test:run -- router           # Run router tests
pnpm test:run -- guardrails       # Run guardrail tests

# E2E tests
pnpm test:e2e -- agents           # Run agent e2e specs

# Smoke tests (requires running server)
npx tsx scripts/smoke-tests/smoke-test-tool-trigger.ts

Rules

• Unit tests must not call external APIs — mock external services
• Smoke tests require a running server — document the setup in the test file
• E2E tests use Playwright and run against the dev server
• Test both happy paths and error cases (missing API keys, invalid input, service unavailable)
• For retrieval tests, assert that the correct tools are invoked AND that citations appear in output
• Guardrail tests should cover both pass and reject cases
• Keep test files focused — one test file per concern (tools, router, guardrails, etc.)