Extend Smoke Coverage

extend-smoke-coverage

The smoke suite exists to catch regressions in published @mastra/* alphas before they reach users. Coverage gaps are how regressions slip through. When a new feature, endpoint, or Studio page lands, the smoke suite needs to grow to match — otherwise the next alpha can break it silently.

This skill is the playbook for adding that coverage cleanly.

When to activate

Activate any time you need to add or update tests in response to:

• A new /api/* route, request shape, or response field landing upstream.
• A new Studio page, tab, or component.
• A new Mastra primitive (agent option, tool builder hook, workflow step kind, processor type, scorer kind, channel, schedule trigger, etc).
• A user asking "do we cover X?" — start here, answer from the COVERAGE docs, then add what's missing.
• A user asking "what's new since the last alpha?" or "what should we add coverage for?" — jump to the Discovering what's missing section below before touching any files.
• A bug post-mortem revealing the smoke suite should have caught it.

Discovering what's missing

When the user asks "what should we cover?" or "is anything new since the last alpha?", don't guess from memory — diff the live truth.

A. Diff the live API surface against existing tests

The smoke fixture exposes GET /api/system/api-schema, which returns the full route catalog the running server actually serves. Compare it against the routes already exercised:

# All routes the running server exposes (requires a built + running fixture)
pnpm build && pnpm dev &   # or however the fixture is up
curl -s http://localhost:4111/api/system/api-schema \
  | jq -r '.routes[] | "\(.method) \(.path)"' | sort -u > /tmp/server-routes.txt

# All routes referenced by current tests
grep -rhoE "'/api/[a-zA-Z0-9/_:{}-]+" tests | sort -u > /tmp/tested-routes.txt

# Routes the server has but tests never touch
comm -23 /tmp/server-routes.txt /tmp/tested-routes.txt | head -40

The leftover list is the candidate set. Skim it, group by route prefix, match against tests/COVERAGE.md's section table to decide if it's a genuine gap or already covered indirectly (e.g. a /:id variant covered by the list test).

B. Diff Studio routes against existing UI specs

Studio's route table lives in packages/playground/src/App.tsx upstream. Fetch it via the debug-mastra-framework recipe (single curl, no clone):

SHA=$(gh api repos/mastra-ai/mastra/commits/main --jq '.sha')
curl -fsSL "https://raw.githubusercontent.com/mastra-ai/mastra/$SHA/packages/playground/src/App.tsx" \
  | grep -E "path:|<Route" > /tmp/studio-routes.txt

# Routes the UI specs visit
grep -rhoE "page\.goto\('/[a-zA-Z0-9/_-]+" tests-ui | sort -u > /tmp/tested-ui-routes.txt

Cross-reference — anything in Studio that no spec visits is a candidate.

C. Scan recent upstream changesets

Changesets describe what's about to ship in the next alpha, named per intent. They're the highest-signal source for "new feature landed":

gh api 'repos/mastra-ai/mastra/contents/.changeset?ref=main' \
  --jq '.[] | select(.name | endswith(".md")) | select(.name != "README.md") | .name' \
  | head -20

# Read any that look feature-shaped
gh api 'repos/mastra-ai/mastra/contents/.changeset/<file>.md?ref=main' \
  --jq '.content' | base64 -d

Filter for feat(...) / fix(...) lines that mention new endpoints, new pages, new agent options, new processor kinds, etc. chore and internal refactors usually don't need new smoke coverage.

D. Scan recent commits since the last tested alpha

For a more exhaustive sweep:

# Find the sha of the alpha currently under test (see debug-mastra-framework SKILL step 3)
LAST_SHA="<resolved sha for the last green alpha>"

gh api "repos/mastra-ai/mastra/compare/$LAST_SHA...main" \
  --jq '.commits[] | "\(.sha[0:10]) \(.commit.message | split("\n")[0])"' \
  | grep -iE '^(.{10}) (feat|fix)\(' \
  | head -40

This shows every user-facing change between the last alpha we tested and main. Inspect commit diffs (gh api repos/mastra-ai/mastra/commits/<sha>) for any that add routes, components, or primitives.

E. Triage what you find

Not everything new needs smoke coverage. Use this rubric:

When in doubt: if a regression here would silently break a published alpha for users, add coverage. If a regression would surface as a TypeScript build error before publish, skip it.

Mental model

There are exactly three files that change together for any coverage addition. Skipping any one is a smell.

1. The test file — tests/<area>/<feature>.test.ts (Vitest, API) or tests-ui/<area>/<feature>.spec.ts (Playwright, UI).
2. The fixture (only if the feature needs server-side state) — register the agent/tool/workflow/etc in src/mastra/<kind>/ and wire it into src/mastra/index.ts.
3. The COVERAGE doc — tests/COVERAGE.md for API, or tests-ui/COVERAGE.md for UI. Update the summary row count, add the feature to the right section, and bump the "last updated" date.

If you only touch (1), the suite grows but nobody can find what's covered. If you only touch (3), the doc lies. If you only touch (2), the fixture is dead code.

Workflow

1. Find the right home

API tests live under tests/ grouped by route prefix:

tests/agents/        →  /api/agents/*
tests/workflows/     →  /api/workflows/*
tests/memory/        →  /api/memory/*
tests/observability/ →  /api/observability/*
tests/stored/        →  /api/stored/*  (CRUD for stored entities)
tests/schedules/     →  /api/schedules/*
...

UI tests live under tests-ui/ grouped by Studio route:

tests-ui/agents/        →  /agents/*
tests-ui/workflows/     →  /workflows/*
tests-ui/observability/ →  /observability/*
tests-ui/cms/           →  /cms/*
...

If the new feature doesn't fit an existing group, create a new directory. Name it after the route prefix or Studio section, not the ticket / PR.

2. Decide: API, UI, or both?

Default to API-first: API tests are fast, deterministic, easy to debug. Add UI only when the feature is user-visible in Studio.

3. Check if the fixture already supports it

Before adding fixtures, grep what's already registered:

grep -r "name: '" src/mastra/agents src/mastra/tools src/mastra/workflows
cat src/mastra/index.ts | head -80   # see what's wired into Mastra

Reuse existing fixtures whenever possible (e.g. test-agent, calculator, helper-agent). New fixtures cost server startup time and DB rows on every run.

4. Add the fixture only if needed

If the new feature requires server-side state (a new tool kind, a new workflow shape, a scorer that exercises a code path nothing else hits), add it under src/mastra/<kind>/ and export from src/mastra/<kind>/index.ts. Then wire it into src/mastra/index.ts's new Mastra({ ... }) call.

Keep fixtures minimal:

• Simplest possible schema that triggers the code path.
• No real LLM calls in fixtures (the test-agent reuses gpt-4o-mini via OpenAI; reuse it instead of adding a new model).
• No setInterval, no every-second cron, no background timers (see KNOWN_ISSUES.md — the scheduler race we already burned days on).

5. Write the test

API (Vitest):

import { describe, expect, it } from 'vitest';
import { fetchApi, fetchJson } from '../utils.js';

describe('<feature> — <surface>', () => {
  it('GET /api/<route> returns <expected shape>', async () => {
    const { status, data } = await fetchJson<any>('/api/<route>');
    expect(status).toBe(200);
    expect(data).toMatchObject({ /* minimum invariants */ });
  });

  it('GET /api/<route>/:id returns a structured 404 for unknown id', async () => {
    const res = await fetchApi('/api/<route>/does-not-exist-smoke');
    expect(res.status).toBe(404);
  });
});

UI (Playwright):

import { test, expect } from '@playwright/test';

test.describe('<Feature>', () => {
  test('<page> renders without console errors', async ({ page }) => {
    const errors: string[] = [];
    page.on('pageerror', err => errors.push(err.message));

    await page.goto('/<route>');
    await expect(page.getByRole('heading', { name: /<feature>/i })).toBeVisible();

    expect(errors, `page errors: ${errors.join('\n')}`).toEqual([]);
  });
});

Conventions that have paid off:

• fetchApi / fetchJson from tests/utils.ts — never use raw fetch.
• page.on('pageerror') in every UI spec — catches the bugs that visible-element assertions miss.
• Assert minimum invariants, not full deep-equals. Upstream is free to add fields; the suite should not break on additive changes.
• /<route> 404 on unknown id — always include a negative case.
• Use helpers.ts (UI) — fillAndSend, waitForAssistantMessage — for any chat interaction. Don't reinvent input timing.
• No setTimeout in tests. Use Playwright/Vitest waiting primitives.
• Tag LLM-touching tests with @llm so the matrix-level retry budget applies (see playwright.config.ts / vitest.config.ts).
• Prefer getByRole(role, { name }) over getByLabel(name) for form controls. Studio's design system often renders a visible labelled <span> next to a hidden <input> carrying the same accessible name — getByLabel then matches both and fails strict mode. getByRole('radio'|'checkbox'|'textbox') only matches the actual control. See "UI locator pitfalls" below.
• Don't anchor on generic semantic wrappers (getByRole('listitem'), getByRole('list')) for table-like rows. Studio renders rows as <button>, <div>, or other elements, not <li>. Scope to the parent region/tabpanel and match the row by accessible name instead.
• Always design failure-safe cleanup. Any test that creates filesystem state, DB rows, or remote registrations needs an afterAll (or afterEach) hook that wipes that state even if the test body throws. See "Failure-safe cleanup" below — this has caused several "phantom" CI failures where the retry sees "already exists" because attempt 1 failed mid-flight.

5a. Probe before you assert

Before writing any assertion, run the endpoint once and capture the real response. Don't write assertions from memory or from what the docs imply — assert against the actual bytes the server emits today.

Cheap, disposable workflow:

// tests/_probe.test.ts — write, run, delete. Do NOT commit.
import { describe, it } from 'vitest';
import { fetchJson } from './utils.js';

describe('probe', () => {
  it('embedders', async () => {
    const r = await fetchJson('/api/embedders');
    console.log('EMBEDDERS', r.status, JSON.stringify(r.data).slice(0, 800));
  });
});

pnpm test -- tests/_probe.test.ts 2>&1 | grep EMBEDDERS
rm tests/_probe.test.ts   # delete before committing

Then write the real test against the response shape you actually saw — deterministic ids, exact error strings, full field-by-field structure.

5b. Banned assertion patterns

The whole point of smoke is to catch regressions. Weak assertions defeat that — they pass against any reasonably-shaped garbage. Reviewers (human and agentic) reject these patterns. If you find yourself writing one, stop and ask "what does this actually prove?"

Concrete examples from real smoke fixes:

// ❌ Was passing against literally any 4xx-or-5xx
expect([404, 500]).toContain(res.status);

// ✅ Exact status + the id appears in the message
expect(res.status).toBe(404);
expect(data.error).toMatch(/does-not-exist-smoke/);

// ❌ Passed even if `output` was empty or `usage` was missing fields
expect(typeof data.id).toBe('string');
expect(data.usage).toBeDefined();

// ✅ Proves the LLM actually produced text and the token math is internally consistent
const msg = data.output.find(o => o.type === 'message');
const text = msg.content.find(c => c.type === 'output_text');
expect(typeof text.text).toBe('string');
expect(text.text.length).toBeGreaterThan(0);
expect(data.usage.total_tokens).toBe(data.usage.input_tokens + data.usage.output_tokens);

5b-passive. Passive-accumulator endpoints (never assert empty)

A whole class of routes return whatever telemetry has accumulated on the running fixture so far. They are not isolated per test — every agent run, workflow execution, feedback POST, or score write the rest of the suite performs leaves a row these endpoints will surface.

Asserting toEqual([]) against any of them is an order-dependent timebomb: it passes when the test runs first, fails the moment another test runs ahead of it. This bit hard during the May 2026 observability expansion — six discovery tests passed in isolation, failed in the full suite once agent/workflow tests populated the trace store ahead of them.

Known passive accumulators (assume more exist — anything reading from OTLP, the score store, or feedback table behaves this way):

• GET /api/observability/discovery/environments
• GET /api/observability/discovery/entity-types
• GET /api/observability/discovery/entity-names
• GET /api/observability/discovery/metric-names
• GET /api/observability/discovery/service-names
• GET /api/observability/discovery/tags
• GET /api/observability/traces, /spans, /logs (list views)
• Anything reading from observability/feedback, observability/scores list endpoints without a filter that uniquely scopes to your test

Rule: don't assert emptiness. Assert (a) the shape — array of the expected scalar type — and (b) a value the fixture always emits.

// ❌ Passes alone, fails in the full suite the moment any other test runs an agent
expect(data.serviceNames).toEqual([]);

// ✅ Catches "returns objects instead of strings" AND "endpoint doesn't query telemetry at all"
expect(Array.isArray(data.serviceNames)).toBe(true);
for (const s of data.serviceNames) expect(typeof s).toBe('string');
expect(data.serviceNames).toContain('smoke-test');   // hardcoded service name in src/mastra/index.ts

Fixture-guaranteed stable values you can lean on:

For an endpoint where you genuinely need an empty result (e.g. metric-label-keys?metricName=does-not-exist), use a unique nonexistent input — that lookup is filtered, not a global accumulator.

5b-nonce. Nonce convention for state-mutating tests

Any POST that writes to a queryable store (feedback, scores, traces, anything aggregations / list endpoints can read back) must tag the row with a per-run nonce. Otherwise two test files querying the same generic value (feedbackType: 'thumbs') race each other — one creates data, the other expects empty, full-suite run goes red.

// At module scope, once per test file:
const FEEDBACK_TYPE = `smoke-<area>-${Date.now()}`;
const SCORER_ID     = `smoke-scorer-${Date.now()}`;
const ENTITY_ID     = `smoke-entity-${Date.now()}`;

// Use the nonce both when writing and when querying back:
await fetchApi('/api/observability/feedback', {
  method: 'POST',
  body: JSON.stringify({ feedbackType: FEEDBACK_TYPE, entityId: ENTITY_ID, ... }),
});

const { data } = await fetchJson<...>('/api/observability/feedback');
const row = data.feedback.find(f => f.feedbackType === FEEDBACK_TYPE);
expect(row).toBeDefined();

Date.now() is the convention because it (a) survives parallel test runs in the same process, (b) is unique across CI retries, and (c) makes leaked rows easy to spot and clean up by prefix.

5b-ui. UI locator pitfalls

Playwright failures in past CI runs that turned out to be smoke-fixture bugs followed two recurring patterns. Avoid both.

Banned: getByLabel(name) on radios, checkboxes, and switches. Studio renders many controls as a visible labelled wrapper + a hidden native input. Both carry the same accessible name, so getByLabel matches two elements and Playwright fails strict mode with resolved to 2 elements.

// ❌ Strict-mode time bomb — breaks the moment Studio adds an inner <input>
await page.getByLabel('Generate').click();
await expect(page.getByLabel('Stream')).toHaveAttribute('data-state', 'checked');

// ✅ Only matches the role-bearing element
await page.getByRole('radio', { name: 'Generate' }).click();
await expect(page.getByRole('radio', { name: 'Stream' })).toHaveAttribute('data-state', 'checked');

Banned: getByRole('listitem') for table rows. Studio renders rows in skills/agents/workflows lists as <button> (with the name, path, and description inline) plus sibling action buttons — not as <li>. Anchoring on listitem quietly stops matching the next time the design system changes.

// ❌ Brittle — assumes a specific wrapper element
const row = page.locator('main').getByRole('listitem').filter({ hasText: skillName });

// ✅ Scope to the panel, match the row by accessible name
const skillsTab = page.getByRole('tabpanel', { name: /Skills/ });
const row = skillsTab.getByRole('button', { name: new RegExp(`^${skillName}\\b`) });

When in doubt, open error-context.md from a failing run's test-results/ directory — its YAML accessibility snapshot tells you the exact roles and names Studio is currently emitting.

5c. Failure-safe cleanup for tests that create state

Tests that mutate the filesystem (/api/workspaces/*/fs/*), register entities, or install skills must wipe that state in an afterAll / afterEach — not just at the end of the test body. If the body throws after creating the state but before the cleanup line, the next CI iteration inherits the leftover. That's how a fixable assertion failure becomes a multi-attempt "already exists" mystery that masks the real bug.

test.describe('Skills install', () => {
  // ❌ Cleanup never runs if any earlier line in the test throws.
  test('install + remove', async ({ request }) => {
    await request.post('/api/.../install', { ... });
    await expect(/* ... */).toBeVisible();           // <-- if this fails, the next
    await request.delete('/api/.../uninstall');       //     line never executes
  });

  // ✅ afterAll runs regardless of test outcome — next iteration starts clean.
  test.afterAll(async ({ request }) => {
    await request.delete(
      `/api/workspaces/test-workspace/fs/delete?path=.agents&recursive=true&force=true`,
    );
  });
});

A self-audit question for any new test: "If line 5 of my test body throws, does the next CI run start with the same on-disk state as before this test existed?" If the answer is no, add the unconditional cleanup before committing.

5d. Self-audit before committing

After writing tests but before running the full suite, run a focused self-audit. The cheapest way is a sub-agent pass with this exact prompt:

Read these new test files: <list of files>. For each assertion, ask: "Would this still pass if the server returned empty, wrong, or structurally-similar-but-garbage data?" Flag every assertion that would pass against bad data. Check specifically for:

• Bare expect(typeof x).toBe(...) as the only check on a field (NOTE: for (const x of arr) expect(typeof x).toBe('string') is acceptable as a shape guard when paired with a separate value/length assertion — flag only if it's the only check on that field)
• [a, b].toContain(status) accepting multiple statuses
• length > 0 / toBeDefined() without follow-up shape assertions
• fetchJson<any>(...) instead of a typed response
• Error bodies asserted only as "is a string"
• 400 responses without issues[].field checks
• try/catch swallowing errors inside test bodies
• Deep-equality of whole bodies (breaks on additive upstream changes)
• toEqual([]) on any passive-accumulator endpoint (see 5b-passive) — these endpoints surface telemetry from every other test, so emptiness is order-dependent and flakes the suite
• State-mutating POSTs that write generic values ('thumbs', 'agent-1') to queryable stores without a per-run nonce (see 5b-nonce). The aggregations/list query in another test file will collide.
• (UI only) getByLabel(...) used on form controls — Studio renders visible labels and hidden inputs as siblings; prefer getByRole.
• (UI only) getByRole('listitem') / getByRole('list') used to anchor table-like rows. Studio rows are usually <button> or <div>; prefer scoping by tabpanel/region and matching by name.
• (UI only) Test creates filesystem or registry state but cleanup lives in the test body (not in afterAll/afterEach). A failure mid-body leaks state to the next CI iteration.

For each finding, name the file, line range, and a concrete tightened assertion.

Don't auto-strip Array.isArray(x) before toEqual([]) as "redundant". It is redundant for the equality check itself, but it's often the only line that produces a useful failure message when the endpoint returns an object instead of an array. Keep it unless you're replacing it with something equally diagnostic.

The audit is a loop, not a single pass. After applying flagged fixes:

1. Re-run the audit on the patched files — fixes can introduce new issues (a toMatchObject that's too loose, a removed guard that was load-bearing).
2. Run the focused tests for the patched files.
3. Run the full suite (pnpm test:all). Order-dependent and cross-file-contamination bugs only surface here.
4. If the full-suite run reveals a failure, that's a real audit finding — patch, then go back to step 1.
5. Commit only when audit + full-suite are both clean in the same iteration.

In the May 2026 observability expansion the first audit pass missed the order-dependent discovery assertions; the full-suite run caught them; the second audit confirmed the fix. The loop is what makes the process reliable.

This pass should also be run by reviewers on any smoke-coverage PR.

6. Run the test locally before committing

# API
pnpm build && pnpm test -- tests/<area>/<feature>.test.ts

# UI
pnpm build:studio && pnpm test:ui -- tests-ui/<area>/<feature>.spec.ts

# Both, full suite (don't skip — order-dependent bugs are real here)
pnpm build:studio && pnpm test:all

If your new test passes alone but the full suite fails, you've hit either (a) a fixture collision (your fixture clashes with another) or (b) an upstream framework bug exposed by the new code path. Check KNOWN_ISSUES.md and the debug-mastra-framework skill.

7. Update COVERAGE.md

Open tests/COVERAGE.md and/or tests-ui/COVERAGE.md and:

1. Bump the header: > N tests across M test files — last updated YYYY-MM-DD.
2. Find the right section row in the Summary table; bump its count and update the Notes column if the addition is non-obvious (new endpoint, new gating, NEW marker for net-new sections).
3. Bump the Total.
4. For API: update the "Coverage by /api/* route group" cross-ref table if you added a new route prefix.
5. For UI: if you added a new Studio surface, note it in the section row.

If your addition uncovers something previously listed as 🔒 (blocked), flip it to ✅ and remove the Notes blocker.

8. Sanity-check the diff

Before committing:

git diff --stat
# Expected: 1 test file (+), maybe 1-2 src/mastra files (+/M), 1-2 COVERAGE.md (M)

If you see changes to playwright.config.ts, vitest.config.ts, smoke.yml, or KNOWN_ISSUES.md, stop and re-justify — those usually shouldn't move when adding coverage. Carve them into a separate commit with reasoning if they're truly needed.

What NOT to do

• Don't add tests for behavior that isn't published yet. Smoke runs against @mastra/*@alpha — testing unreleased features makes the matrix fail until the alpha catches up.
• Don't add fixtures "just in case". Every fixture costs server startup time and migration rows. Reuse before adding.
• Don't deep-equal whole response bodies. Upstream adds fields all the time. Match the minimum shape.
• Don't bump COVERAGE.md numbers without running pnpm test. The count in the doc must match the actual test count.
• Don't skip the negative test (404 / 4xx). A regression that flips a 404 to a 500 is exactly what smoke is for.
• Don't introduce real-time triggers (setInterval, every-second cron, watch loops) — see KNOWN_ISSUES.md section 1. They poison the suite for everyone.
• Don't add a UI test without page.on('pageerror', ...). Visible assertions miss silent client-side throws; the pageerror listener is the only reliable canary.
• Don't skip the full-suite run. Single-file passes don't catch order-dependent or schema-poisoning regressions.
• Don't ship weak assertions. See section 5b. typeof x === 'string', [200, 500].toContain(status), length > 0 alone, toBeDefined() without drill-down, and fetchJson<any> all pass against broken servers and are rejected in review.
• Don't skip the self-audit in section 5d. New test files get a pass over them before commit, and another pass after any fix — it's a loop, not a single shot. The full-suite run is part of the loop.
• Don't assert toEqual([]) on telemetry/discovery endpoints. They surface accumulated state from every other test in the suite. Assert shape + a fixture-guaranteed value (see 5b-passive).
• Don't write generic values to telemetry stores from a test. Use a per-run nonce (`smoke-<area>-${Date.now()}`) so other tests querying the same store don't collide with your row (see 5b-nonce).
• Don't use getByLabel on radios/checkboxes/switches. Studio renders visible-label + hidden-input pairs; getByLabel matches both and fails strict mode. Use getByRole('radio'|'checkbox', { name }). See 5b-ui.
• Don't anchor on getByRole('listitem') for table rows. Studio rows are <button>/<div>, not <li>. Scope by parent region and match by accessible name. See 5b-ui.
• Don't put cleanup at the end of a test body. It won't run if any earlier line throws, and the next CI iteration inherits the leftover state. Use test.afterAll / test.afterEach. See 5c.