Maestro Smoke Test Contract

Maestro Smoke Test Contract

Defines the artifact schema for Maestro E2E test verification. The stop hook validates these artifacts exist and tests passed.

Purpose

Maestro smoke tests prove the mobile app functions correctly after a fix. They verify actual user flows on simulator/emulator.

Required Artifacts

After Maestro verification, the following must exist:

.claude/maestro-smoke/
├── summary.json          # Pass/fail + test results (REQUIRED)
├── screenshots/          # Test screenshots (REQUIRED)
│   ├── *.png
│   └── ...
└── output.log            # Maestro test output (optional)

summary.json Schema

{
  "passed": true,
  "tested_at": "2026-01-28T10:00:00Z",
  "tested_at_version": "abc1234",
  "platform": "ios",
  "device": "iPhone 15 Pro Simulator",
  "app_id": "be.revonc.mobileapp",
  "maestro_version": "1.38.0",
  "flows_executed": [
    {
      "name": "J2-returning-user-login.yaml",
      "path": ".maestro/journeys/J2-returning-user-login.yaml",
      "passed": true,
      "duration_ms": 45000,
      "screenshots": [
        "j2_01_app_launched.png",
        "j2_07_stepping_stones_visible.png"
      ]
    }
  ],
  "total_flows": 1,
  "passed_flows": 1,
  "failed_flows": 0,
  "total_screenshots": 8,
  "error_message": null
}

Flow Object Schema

Pass Conditions

The stop hook requires ALL of these:

1. passed: true
2. total_flows >= 1 (at least one flow executed)
3. failed_flows == 0
4. tested_at_version == current_git_version (not stale)
5. Screenshots directory exists and contains files

Checkpoint Integration

The checkpoint must include Maestro test fields:

{
  "self_report": {
    "maestro_tests_passed": true,
    "maestro_tests_passed_at_version": "abc1234"
  },
  "evidence": {
    "maestro_flows_tested": [
      "J2-returning-user-login.yaml"
    ],
    "screenshots": [
      ".claude/maestro-smoke/screenshots/j2_07_stepping_stones_visible.png"
    ],
    "platform": "ios",
    "device": "iPhone 15 Pro Simulator"
  }
}

Staleness Detection

Artifacts become stale when code changes after verification:

tested_at_version: abc1234
current_version:   def5678  <- Different!
-> STALE: Must re-verify

The stop hook automatically rejects stale artifacts and instructs re-verification.

Creating Artifacts

After Successful Maestro Run

#!/bin/bash
# create-maestro-artifacts.sh

mkdir -p .claude/maestro-smoke/screenshots

# Copy Maestro screenshots
cp -r .maestro/screenshots/* .claude/maestro-smoke/screenshots/ 2>/dev/null || true

# Get test info
VERSION=$(git rev-parse --short HEAD)
TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
PLATFORM="${1:-ios}"
DEVICE="${2:-iPhone 15 Pro Simulator}"

# Create summary
cat > .claude/maestro-smoke/summary.json << EOF
{
  "passed": true,
  "tested_at": "$TIMESTAMP",
  "tested_at_version": "$VERSION",
  "platform": "$PLATFORM",
  "device": "$DEVICE",
  "app_id": "be.revonc.mobileapp",
  "flows_executed": [
    {
      "name": "J2-returning-user-login.yaml",
      "path": ".maestro/journeys/J2-returning-user-login.yaml",
      "passed": true,
      "screenshots": $(ls .claude/maestro-smoke/screenshots/*.png 2>/dev/null | jq -R -s 'split("\n") | map(select(length > 0) | split("/") | .[-1])')
    }
  ],
  "total_flows": 1,
  "passed_flows": 1,
  "failed_flows": 0
}
EOF

echo "Maestro artifacts created in .claude/maestro-smoke/"

After Failed Maestro Run

#!/bin/bash
# create-maestro-failure-artifacts.sh

mkdir -p .claude/maestro-smoke/screenshots

# Copy any screenshots captured before failure
cp -r .maestro/screenshots/* .claude/maestro-smoke/screenshots/ 2>/dev/null || true

# Get info
VERSION=$(git rev-parse --short HEAD)
TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
ERROR_MSG="${1:-Test failed}"

# Create summary with failure
cat > .claude/maestro-smoke/summary.json << EOF
{
  "passed": false,
  "tested_at": "$TIMESTAMP",
  "tested_at_version": "$VERSION",
  "platform": "ios",
  "device": "iPhone 15 Pro Simulator",
  "app_id": "be.revonc.mobileapp",
  "flows_executed": [
    {
      "name": "J2-returning-user-login.yaml",
      "path": ".maestro/journeys/J2-returning-user-login.yaml",
      "passed": false,
      "error": "$ERROR_MSG"
    }
  ],
  "total_flows": 1,
  "passed_flows": 0,
  "failed_flows": 1,
  "error_message": "$ERROR_MSG"
}
EOF

echo "Maestro failure artifacts created in .claude/maestro-smoke/"

Minimum Required Flows

For a quick smoke test, at minimum run:

1. J2-returning-user-login.yaml - Verifies login and main app access

For more thorough verification:

1. J2 - Login flow
2. J3 - Navigation
3. J5 - Profile/settings

For complete verification:

• All journeys (J1-J5)
• Negative tests (N1-N3)

Screenshot Requirements

Screenshots must capture:

1. Initial state - App launched
2. Post-login - Main screen visible
3. Key UI elements - Stepping stones, profile, etc.

Naming convention: {journey}_{step}_{description}.png

Examples:

• j2_01_app_launched.png
• j2_04_login_screen.png
• j2_07_stepping_stones_visible.png

Relationship to Unit Tests

Both are required. Unit tests catch logic errors; Maestro tests catch integration and UI issues.