Ever Test Gen

ever-test-gen (v1)

Characterization test generator for the EVO agent ecosystem (S032 WP-7).

Status: PRODUCTION (S032 WP-7 ship). Design document: /path/to/project/docs/plans/2026-05-13-evo-agent-design.md §4.1 + §9. Composes: integration-flow-testing@1.x (does NOT extend or fork — uses its test-generation primitives as a library).

What this skill does

Generates characterization tests — tests that capture the legacy codebase's observed behaviour for use during a version-upgrade or CVE-fix.

Three test types per mode:

Mode-a (intent-map-only) NEVER triggers this skill — read-only analysis.

Mandatory confidence headers

Every generated test file has this header at the top:

"""
EVO-generated test — CONFIDENCE: characterization-aid
This test was auto-generated by Evo from the intent map + api_delta.
It captures observed legacy behaviour, NOT a correctness specification.
Requires user review before being relied on for refactor safety.

Source seed: <reference into intent-map.test_seeds[]> | <api_delta line> | <CVE record>
Wiring evidence: <snapshot_hash> @ <symbol> @ <site>
"""

import pytest

@pytest.mark.evo_generated
@pytest.mark.evo_confidence("characterization-aid")
@pytest.mark.evo_requires_review
def test_...():
    ...

(JavaScript version uses describe.skipIf + JSDoc.)

Hard rules (HARD-RULE alignment per design §13)

1. HARD-RULE 1 — NEVER writes claim files (CB4). Bob issues, skill heartbeats.
2. HARD-RULE 1 — NEVER writes the integration ledger. Emits one transition request per run.
3. HARD-RULE 2 — Bug-for-bug compatibility. Migration-confirmation tests treat legacy output as oracle. If legacy returns wrong output for input X, the upgraded code MUST return the same wrong output.
4. NEVER overwrites existing user tests. Generated paths are namespaced test_evo_<mode>_<comp>__<symbol>__<type>.py|js — collision check before write. Refuse + advisory on collision.
5. Always writes confidence + requires-user-review header. No exceptions.
6. Composes integration-flow-testing. Imports its symbol from the sibling skill; does NOT extend/fork the SKILL.md or scripts.
7. Heartbeats every 60s. Calls claims.heartbeat_claim(claim_uuid).
8. Schema validation on every test seed. Reads functional-intent.v1 files and validates test_seeds[] before consuming.

What lives in scripts/

• run.py — CLI + dispatch
• regression_replay.py — emit baseline replay tests (pytest + jest)
• migration_confirm.py — emit api_delta-aware migration tests (oracle = legacy)
• cve_proof.py — emit CVE-fix proof tests (mode-c only)
• test_header.py — mandatory header generator + pytest.mark metadata
• compose_iflow.py — adapter that calls integration-flow-testing primitives

What lives in references/

• bug-for-bug-oracle.md — how the differential-snapshot oracle is built
• cve-proof-anatomy.md — what a CVE proof-of-fix test actually contains
• composition-not-extension.md — why ever-test-gen never imports IFT internals

What lives in templates/

• pytest-regression.py.tpl — Python pytest template
• pytest-migration.py.tpl — Python pytest migration template
• pytest-cve-proof.py.tpl — Python CVE-fix proof template
• jest-regression.js.tpl — JavaScript jest template
• jest-migration.js.tpl — JavaScript jest migration template
• jest-cve-proof.js.tpl — JavaScript CVE-fix proof template

Determinism

Same inputs (intent-map + api_delta + plan.yaml) → byte-identical test files. The skill does NOT consult an LLM; it's pure templating.

How to invoke

python3 ~/.claude/skills/ever-test-gen/scripts/run.py \
  --project-root <abs path> \
  --run-id <uuid> \
  --claim-uuid <uuid> \
  --plan-path <path-to-plan.yaml> \
  --intent-map-path <path-to-intent-map.yaml> \
  --mode version-upgrade \
  [--api-delta-path <path>] \
  --language-target python  # or javascript