Testing

Testing — Strategy & Philosophy

Testing is not verification after the fact. It is feedback on design. A test that is hard to write reveals coupling in production code. A test that is hard to name reveals unclear thinking about behavior. The test suite is the living specification of the system — what the code is actually supposed to do. That's true whether a human or an AI agent wrote the code.

Core Beliefs

The Test Portfolio

The test pyramid describes a useful default distribution, not a law. Unit tests form the base because they are fast and precise. Higher-level tests are valuable because they exercise wiring, collaboration, and user journeys. The portfolio rule is: choose the cheapest layer that gives the scenario the confidence it needs.

When a high-level test finds a bug, reproduce it at the lowest useful level before fixing it. The focused regression test is the permanent guardrail; the high-level test is the discovery mechanism.

Decision Guide

Testing With AI Coding Agents

Strict red/green/refactor exists to manage human short-term memory. Agents have the opposite profile — large working memory, but prone to hallucination and to gaming the test rather than satisfying it. The cadence and trust model both need to adapt.

For the full pipeline and trust model, see Testing With AI Coding Agents. For LLM applications, also read Martin Fowler's Testing Canon.

Testing Anti-Patterns

Integrated Example

Before — hard to test (the design is the problem):

function sendOverdueReminders() {
  const users = db.query("SELECT * FROM users WHERE balance < 0");
  for (const user of users) {
    if (Date.now() - user.lastReminded > WEEK) {
      emailClient.send(user.email, "You owe us money");
    }
  }
}

This function forces a test to control a database, a clock, and an email client. The business rule — who should be reminded — is buried inside the side effects.

Diagnosis: push I/O to the edges and keep the rule pure.

After — pure core, I/O at the edge:

function usersToRemind(users, now) {
  return users.filter(
    (user) => user.balance < 0 && now - user.lastReminded > WEEK,
  );
}

function sendOverdueReminders() {
  for (const user of usersToRemind(db.allUsers(), Date.now())) {
    emailClient.send(user.email, "You owe us money");
  }
}

usersToRemind([...], fixedNow) is a one-line unit test with no doubles. The remaining shell is a thin integration concern. Hard-to-test became easy-to-test by changing the design, not by adding mocks.

Read On Demand

Specialist Skills

Benchmark

Scenario: .benchmarks/scenarios/testing-001-agent-test-strategy.md · Run: 2026-08-31 · Log: .benchmarks/runs/2026-08-31/testing-001-agent-test-strategy.json

PASS (run 2026-08-31). Opus and haiku +33; sonnet at ceiling. Gate per .agents/skills/skill-optimizer/rules/release-gates.md.