# Site Tests

> Use when working on documentation tests (site-tests module). Covers test generation pipeline, fixtures, assertions, and debugging.

- Skill: `tomevault-io/site-tests` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/site-tests`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/site-tests/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/site-tests

---


# Site Tests — Documentation Test Framework

## Pipeline
```
MDX docs (CurlTest/TestScenario components)
  → Astro build extracts test data
  → site-tests/target/generated-test-resources/test-scenarios.json
  → TestGenerator.java creates .feature file
  → site-tests/target/generated-test-resources/features/documentation-tests.feature
  → Cucumber runs tests
```

**DO NOT edit** `documentation-tests.feature` directly — it's auto-generated. Edit the MDX source files instead.

## Run Tests
```bash
./mvnw test -pl site-tests -Psite-tests
```
Redirect output to file and grep for errors (output is very long).

## MDX Test Components

**TestScenario** wraps a group of CurlTests for a checkpoint:
```mdx
<TestScenario checkpoint="quarkus/examples/doc-checkpoints/03-with-history">
  <CurlTest steps={`...`}>```bash
  curl ...
  ```</CurlTest>
</TestScenario>
```

**CurlTest** `steps` prop contains Cucumber step definitions:
- `Then the response status should be 200`
- `And the response body should be json:` (strict JSON equality via Jackson `JsonNode.equals()`)
- `And the response should contain "text"` (loose substring check)
- `And the response should match pattern "\\d+"` (regex)

## JSON Assertions

Use `%{response.body.field}` for dynamic values (IDs, timestamps). Use static values for deterministic fields (titles, userIds, accessLevels).

```
And the response body should be json:
"""
{
  "id": "%{response.body.id}",
  "title": "My Static Title",
  "ownerUserId": "bob",
  "createdAt": "%{response.body.createdAt}"
}
"""
```

**Array matching is strict** — the expected JSON must have the exact same number of array elements as the actual response. If other test scenarios create data visible to the current test, include those items in the assertion.

## OpenAI Mock (WireMock)

- Fixtures in `site-tests/openai-mock/fixtures/{quarkus|spring}/{checkpoint-name}/`
- Sequential state machine: `001.json` → `002.json` → ... (matched by scenario state, NOT request content)
- Fixtures are loaded per checkpoint via `DockerSteps.loadFixturesForCheckpoint()`
- WireMock resets between checkpoints

## Shared State Gotcha

All test scenarios share ONE memory-service instance. Conversations created in one scenario are visible in others. Use different conversation IDs across tutorials, or write assertions that tolerate extra data.

## Curl Retry Logic

`CurlSteps.java` retries on 404, 500, 503 (transient startup errors). Max 5 retries, 3s delay.

## Key Files
- `site/src/components/TestScenario.astro` — extracts tests from MDX during Astro build
- `site/src/components/CurlTest.astro` — wraps curl commands with hidden test steps
- `site-tests/src/main/java/.../TestGenerator.java` — generates .feature from JSON
- `site-tests/src/test/java/.../steps/CurlSteps.java` — curl execution + assertions
- `site-tests/src/test/java/.../steps/CheckpointSteps.java` — builds/starts/stops apps
- `site-tests/src/test/java/.../steps/DockerSteps.java` — docker compose + WireMock management

---
> Converted and distributed by [TomeVault](https://tomevault.io/claim/chirino) — claim your Tome and manage your conversions.
<!-- tomevault:4.0:skill_md:2026-04-13 -->