macos-e2e-scaffold
Manual-invocation skill that bootstraps XCUITest infrastructure for macOS SwiftUI projects.
Phase 0 — Self-check
Before any other action, run three refuse-conditions. Any failure → return early with explicit message; no files modified.
| Check | Detect via | Refuse-message |
|---|---|---|
| Swift project | *.xcodeproj directory or Package.swift in cwd |
"Not a Swift project. /macos-e2e-scaffold requires .xcodeproj or Package.swift in project root." |
| SwiftUI macOS app | macOS-discriminating signal (see below) AND SwiftUI scene (grep WindowGroup|Window(|Settings {|MenuBarExtra( in *.swift under source root) |
"No SwiftUI macOS app target detected. Skill is macOS-only — for iOS use /ios-e2e-scaffold, for AppKit use /appkit-e2e-scaffold (deferred)." |
| Not already scaffolded | any directory matching *UITests/ at cwd — EXCLUDING the iOS sibling's *iOSUITests/ (which must not block macOS scaffolding on multiplatform projects) — contains > 1 *.swift (find . -maxdepth 2 -type d -name '*UITests' ! -name '*iOSUITests' then count) — do NOT assume the <App>-prefixed name, since the scheme is not detected until Step 2 |
"UI test target already exists (<found-dir>/, N test files). Skill won't overwrite — extend manually instead." |
macOS-discriminating signal — REQUIRED
WindowGroup is cross-platform (shared with iOS) and does NOT identify macOS on its own. Detect macOS via the FIRST that matches:
- SPM:
Package.swiftcontains.macOS(in aplatforms:clause (and the target is an app, not a library). - xcodegen:
project.yml/xcodegen.ymltarget hasplatform: macOS. - plain
.xcodeproj:project.pbxprojcontainsSDKROOT = macosxorSUPPORTED_PLATFORMSincludesmacosxfor the app target's build config. - Corroborating (necessary-not-sufficient): a macOS-only scene type present (
MenuBarExtra(,Settings {,Window() AND iOS-only signals absent (SDKROOT = iphoneos,platform: iOS,import UIKit,.fullScreenCover().
A project with ONLY iOS signals must be refused with the row's refuse-message pointing to /ios-e2e-scaffold — do NOT scaffold macOS tests from a bare WindowGroup.
Multiplatform targets (.iOS( AND .macOS( in Package.swift, or SUPPORTED_PLATFORMS lists both iphoneos and macosx): Phase 0 passes (macOS is among the platforms) and emits a note — "Multiplatform target detected; scaffolding macOS tests. Run /ios-e2e-scaffold separately for the iOS surface." Treat macosx/.macOS( presence as sufficient; do not refuse just because iOS is also supported.
TARGET_DIR convention
Set once in Phase 0 and used everywhere below (Step 10 file naming, the xcodegen/Xcode UI-test target name, and the runner's -only-testing: argument):
- Single-platform macOS target →
TARGET_DIR = <App>UITests - Multiplatform target →
TARGET_DIR = <App>macOSUITests— so the macOS and iOS UI-test targets coexist./ios-e2e-scaffolduses<App>iOSUITestsin the same situation, and each scaffold's "already scaffolded" glob ignores the sibling's suffixed directory; without the suffix the two would collide on the same directory and the samexcodegen.yml/project.pbxprojUI-test target.
Always emit Phase 0 result on success:
## /macos-e2e-scaffold Phase 0
✅ Swift project detected (<project>.xcodeproj | Package.swift)
✅ SwiftUI macOS app (macOS signal: <SDKROOT=macosx | .macOS( | platform: macOS | macOS-only scene>; scene <Type> in <File.swift>:<line>)
✅ No existing UI test target
Project type: <xcodegen-managed | SPM-based | plain .xcodeproj>
Scheme: <SchemeName>
Source root: <path>
Total .swift files in source root: <N>
[Multiplatform note, if applicable — TARGET_DIR = <App>macOSUITests]
Proceeding with audit + scaffold.
What this skill does
- Audits the project: walks the SwiftUI Scene tree, ranks views by interactive-control density, identifies top 5.
- Suggests accessibility identifiers for each control in top 5 views; applies them after user batch-confirmation.
- Generates ranked TIER-1/2/3 test stubs with
XCTFail("not implemented")placeholders, an identifier-convention doc, and a Claude-readable xcresult runner script.
What this skill is NOT
- Not a review skill. Does not analyse spec/plan/PRD artefacts. Use
/pitfall-verification(will-it-work?),/quality-review(will-it-feel-premium?), or/macos-native-review(is-it-Apple-native?) for those. - Not a code-quality reviewer. Does not check view-code idioms (use Antoine van der Lee's
swiftui-expert-skill) or unit-test idioms (useswift-testing-expert). - Not iOS-aware. Use
/ios-e2e-scaffold. - Not AppKit-aware. Use
/appkit-e2e-scaffold(deferred). - Not snapshot-aware. Use
/swiftui-snapshot-scaffold(deferred). - Not auto-invoked. Manual
/macos-e2e-scaffoldonly — same model assetup-routing. Normally reached via/e2e-route.
Heuristic process (deterministic, Read+Grep based)
Step 1: Detect project type
- If
xcodegen.ymlorproject.ymlexists → xcodegen-managed - Else if
Package.swiftcontains.executableTarget(name:→ SPM-based - Else if
*.xcodeprojexists → plain .xcodeproj
Step 2: Detect scheme name
- xcodegen: read
name:field at root ofxcodegen.yml/project.yml - SPM: read
name:fromPackage(name: ...) - plain .xcodeproj: parse
*.xcodeproj/xcshareddata/xcschemes/*.xcschemefilenames; fallback to project directory name
Step 3: Find source root
- xcodegen: read
targets.<schemename>.sources.pathfrom yml - SPM:
Sources/<TargetName>/ - plain .xcodeproj: parse
project.pbxprojfor main app target's source grouppath =; fallback to<schemename>/
Step 4: Walk Scene tree
- Grep source root:
grep -rn -E 'WindowGroup|Window\(|Settings \{|MenuBarExtra\(' --include='*.swift' - For each Scene file, grep its body for
NavigationLink(destination:,.sheet(content:,.fullScreenCover(content: - Recursively follow destinations to build view-graph (max depth: 5; cycle detection via view-name set)
- For each view in graph, count interactive controls using word-boundary patterns:
\bButton\b\s*[({]\bToggle\b\s*\(\bTextField\b\s*\(\bPicker\b\s*\(\bNavigationLink\b\s*\(
Step 5: Rank views
Sort views by (reference_count + interactive_control_count) descending. Tie-breaker: alphabetical by source-file name, then by line number. Top 5 receive identifier suggestions.
Step 6: Detect TIER mappings
- TIER-1 #1 (Smoke): always (uses Scene-root window-title)
- TIER-1 #2 (Happy-path): pick top-ranked view's primary button. Heuristic:
ButtoncontainingawaitOR action calling a method namedgenerate*/create*/save*/run*/start*. Fallback if no Button matches: pick the first Button in the top-ranked view (by line number); mark the generated stub with comment// HEURISTIC: generic fallback — no save/create/await action matched. Verify this is the right primary action.so user knows to double-check. - TIER-1 #3 (Error-recovery): pick first view (alphabetical by source-file name, then by line number — deterministic) containing
.alert(...),errorMessage,failure, orerror: Error - TIER-2 (Modal): only if
.sheet(isPresented:or.fullScreenCover(isPresented:found in walked tree - TIER-2 (Menubar): only if
.commands { ... }orMenuBarExtra(found - TIER-3 (Multi-window): only if
WindowGroup-count +Window(-count > 1 - TIER-3 (Toolbar): only if
ToolbarItem(count ≥ 2
Step 7: Generate identifier suggestions
For each control in top 5 views:
- Skip controls that already have
.accessibilityIdentifier(...)set — check next 5 lines after the control declaration. Already-identified controls listed in report under "Already identified (preserved)" but not re-suggested. - Skip controls inside
#Preview { ... }blocks orPreviewProvider(static var previews:) conformances — track brace-depth from#Previeworstatic var previewsdeclarations; exclude when depth > 0. - Construct ID as
<ViewName>_<ControlType>_<Purpose> - Purpose extracted from button label, action method name, or property name (in priority order)
- snake_case all parts;
_separator - Examples:
PlanCardView_Button_GeneratePlan,SettingsView_Toggle_EnableTelemetry,AIChatView_TextField_PromptInput
Step 8: Emit suggestions table for user confirmation
Present batch table in markdown:
| File:line | Current code | Suggested identifier |
|---|---|---|
| PlanCardView.swift:34 | Button("Generate") { ... } | PlanCardView_Button_GeneratePlan |
| ... | ... | ... |
Ask: "Apply all N suggestions? [a]ll / [c]herry-pick / [s]kip"
If [c]herry-pick: follow up with one question per suggestion: "Apply suggestion k of N? [y/n]". Accumulate accepted set; apply only that subset in Step 9.
If [s]kip: skip Step 9 entirely; test files in Step 10 still generated, with placeholder identifier comments showing what user must fill in manually.
Step 9: Apply identifiers
Use Edit tool, one identifier per Edit call. On uniqueness conflict (same ID would land on two distinct controls): skip both; flag for manual review in report.
Step 10: Generate test files
Per TIER, write one .swift file with XCTFail("not implemented — fyll inn assertion") placeholder + TODO-comment pointing to source-file:line + suggested assertions in comments.
File naming (all paths use the Phase 0 TARGET_DIR — <App>UITests single-platform, <App>macOSUITests multiplatform):
<TARGET_DIR>/SmokeTest.swift(TIER-1 #1)<TARGET_DIR>/HappyPathTests.swift(TIER-1 #2)<TARGET_DIR>/ErrorRecoveryTests.swift(TIER-1 #3)<TARGET_DIR>/ModalAndMenuTests.swift(TIER-2 if any)<TARGET_DIR>/MultiWindowAndToolbarTests.swift(TIER-3 if any)
Step 11: Generate runner script
Write scripts/run-uitests.sh per template in §Runner-script-template (substitute <APP> with detected scheme name and <TARGET_DIR> with the Phase 0 TARGET_DIR). Make executable: chmod +x scripts/run-uitests.sh.
Step 12: Generate identifier convention doc
Write docs/accessibility-identifiers.md with the convention, examples, rationale, and a table listing all applied identifiers with their source-file:line.
Step 13: Emit final report
Per Output-format section below.
TIER rubric
| Tier | Always-generate? | Heuristic trigger | Test file |
|---|---|---|---|
| 1 #1 Smoke | yes | always | SmokeTest.swift |
| 1 #2 Happy-path | yes | top-ranked view + primary action | HappyPathTests.swift |
| 1 #3 Error-recovery | yes | first .alert/error-state view (alphabetical+line tiebreak) |
ErrorRecoveryTests.swift |
| 2 Modal | conditional | .sheet(isPresented: or .fullScreenCover(isPresented: present |
ModalAndMenuTests.swift |
| 2 Menubar | conditional | .commands { ... } or MenuBarExtra( present |
ModalAndMenuTests.swift |
| 3 Multi-window | conditional | Scene-count > 1 | MultiWindowAndToolbarTests.swift |
| 3 Toolbar | conditional | ToolbarItem(-count ≥ 2 |
MultiWindowAndToolbarTests.swift |
TIER-1 is non-negotiable. Even on a project where heuristics return weak matches, three test files appear. Smoke validates app launches. Happy-path and Error-recovery may need user-tuning but provide a starting structure.
TIER-2 and TIER-3 conditional. Skipped silently if pattern not detected. Report says e.g. "TIER-2 sheet/modal: not generated (no .sheet(isPresented:) found)".
Identifier convention
Format: <ViewName>_<ControlType>_<Purpose>
- snake_case all parts
_separator (consistent grep-by-view:PlanCardView_matches all PlanCardView-controls)- Stable across label-text changes (refactor-safe)
Examples:
PlanCardView_Button_GeneratePlanSettingsView_Toggle_EnableTelemetryAIChatView_TextField_PromptInputToolbarView_NavigationLink_OpenSettings
Convention doc: docs/accessibility-identifiers.md (auto-generated by skill, includes table of all applied identifiers + rationale).
Output format
After Phase 0 emission and identifier-application, the final report (single message at end of skill execution):
## /macos-e2e-scaffold report — <ProjectName>
### Phase 0
✅ Swift project (<project-type>)
✅ SwiftUI macOS app (<N> Scenes detected, macOS signal: <signal>)
✅ No existing UI test target
### Project context
- Scheme: <SchemeName>
- Source root: <path>
- Top 5 views by control density: <View1>, <View2>, <View3>, <View4>, <View5>
### Identifier suggestions (<N> total)
✅ Applied (<N>): <Id1>, <Id2>, ...
⏭️ Skipped on uniqueness conflict (<M>): <Id-X> (collides with <Id-Y> — review manually)
ℹ️ Already identified (preserved <K>): <Id-existing-1>, ...
🚫 Excluded from Preview blocks (<L>)
### Test stubs generated (<S>)
**TIER-1 (3 — must implement)**
- `SmokeTest.swift` :: testTIER1_AppLaunches
- `HappyPathTests.swift` :: testTIER1_<HappyPathName>
- `ErrorRecoveryTests.swift` :: testTIER1_<ErrorPathName>
**TIER-2 (<n> — should implement if applicable)**
- `ModalAndMenuTests.swift` :: testTIER2_<...> (if pattern detected)
**TIER-3 (<n> — patterns not detected | generated)**
- Multi-window: <generated | not generated (single WindowGroup)>
- Toolbar: <generated | not generated (only N ToolbarItem)>
### Runner script
- `scripts/run-uitests.sh` — `xcodebuild test -only-testing:<TARGET_DIR>`, parses xcresult to Claude-readable JSON (Xcode 16+ format; falls back to plaintext on older Xcode)
### Convention doc
- `docs/accessibility-identifiers.md` — `<ViewName>_<ControlType>_<Purpose>`, snake_case, examples, full identifier table
### Project-type integration
<branch-specific instructions per Project-type-specific-behavior section>
### Next steps
1. <project-type-specific build step>
2. `./scripts/run-uitests.sh` — all <S> TIER stubs will fail with `XCTFail("not implemented")`. That's expected — fill in assertions per stub.
3. Re-invoke /macos-e2e-scaffold if you add new top-level views or controls. Skill detects existing UI test target and refuses overwrite (extend manually).
Project-type-specific behavior
xcodegen-managed
Modify xcodegen.yml (or project.yml) to add new target:
targets:
<TARGET_DIR>:
type: bundle.ui-testing
platform: macOS
sources:
- <TARGET_DIR>
dependencies:
- target: <App>
Skill writes the diff. User runs xcodegen generate to regenerate .xcodeproj. Report says: "Run xcodegen generate before opening Xcode."
SPM-based
SwiftPM does NOT support UI Test bundles natively (only .testTarget for unit tests). UI Tests require .xcodeproj.
Skill detects this case:
- Generate test files in
Tests/<TARGET_DIR>/directory - Print warning: "SPM doesn't support UI Test bundles. Generated files exist but require .xcodeproj. Recommend: switch to xcodegen-managed project, or add .xcodeproj manually."
- Refuse to attempt project-modification
This is an honest limitation, not a skill failure. (Note: SPM-only is still a valid input — the skill proceeds and generates files; it does not refuse in Phase 0.)
Runner for SPM-only: do NOT emit the normal Step 11 runner — without an .xcodeproj there is no scheme, so xcodebuild test -scheme "$SCHEME" fails at scheme resolution, not at test execution. Instead write a stub scripts/run-uitests.sh that is honest about the precondition:
#!/usr/bin/env bash
# Auto-generated by /macos-e2e-scaffold (SPM-only project).
echo "SPM-only project: no .xcodeproj → no scheme for 'xcodebuild test'." >&2
echo "Convert to an xcodegen-managed project or add an .xcodeproj, then re-run /macos-e2e-scaffold." >&2
exit 1
So a CI consumer that follows /e2e-route's "Next action: ./scripts/run-uitests.sh" fails fast with a clear reason instead of an opaque scheme-resolution error.
plain .xcodeproj (no xcodegen)
Skill cannot reliably modify project.pbxproj programmatically (one wrong line corrupts the project).
- Generate files in
<TARGET_DIR>/directory - Emit step-by-step manual instructions:
1. Open <App>.xcodeproj in Xcode 2. File > New > Target > macOS > UI Testing Bundle 3. Name: <TARGET_DIR> ← MUST equal the Phase 0 TARGET_DIR EXACTLY. The generated scripts/run-uitests.sh hardcodes `-only-testing:"${UITEST_TARGET}"` with this name baked in; a different Xcode-suggested name makes the runner report "no tests / target not found". 4. Drag generated .swift files into target 5. Set Host Application: <App> 6. Build target once to verify - Report says: "Generated files exist; manual Xcode steps required for target setup. Name the UI-test target exactly
<TARGET_DIR>to match the runner."
Failure modes
| Mode | Detection | Resolution |
|---|---|---|
| Project doesn't build | xcodebuild build fails before scaffold |
Skill stops; user fixes build first |
| Identifier uniqueness conflict | Same ID for 2+ controls | Skip both; flag for manual review |
| Existing UI test target | Phase 0 globs *UITests/ (any dir, excluding *iOSUITests/) *.swift count > 1 — name-agnostic, before scheme detection |
Refuse; suggest manual extension |
| Phase 0 fails | Refuse-condition triggered | Return early; never modify files |
| Platform ambiguous (WindowGroup only) | No macOS-discriminating signal found | Refuse "No SwiftUI macOS app target detected" — do NOT assume macOS from WindowGroup |
| Multiplatform target | .iOS( AND .macOS(, or both platforms in SUPPORTED_PLATFORMS |
Pass; scaffold macOS tests into <App>macOSUITests/; note iOS surface needs /ios-e2e-scaffold |
| User declines identifier-application | [s]kip answer |
Skip Step 9; generate test files with placeholder comments |
| Cherry-pick rejected per-suggestion | User says [n] |
Apply only confirmed subset |
| xcodegen not in PATH | xcodegen not found |
Emit instruction: brew install xcodegen |
| Empty SwiftUI app | Step 4 yields zero interactive controls | Generate Smoke test only; report "No interactive controls found — only Smoke test generated. Add controls and re-invoke." |
Existing .accessibilityIdentifier(...) |
Step 7 next-5-lines check | Preserve; report under "Already identified (preserved)" |
Controls in #Preview { ... } / PreviewProvider |
Step 7 brace-depth tracking | Exclude from suggestions and density count |
| xcresulttool API mismatch | xcrun xcresulttool exits non-zero |
Runner falls back to tail -50 of plaintext xcodebuild output; header notes Xcode 16+ requirement |
| xcodegen.yml unknown structure | Cannot find targets:/name: keys |
Switch to plain .xcodeproj branch; do NOT modify yml; flag ambiguity in report |
Skill never silently corrupts project files. All modifications confirmed; uniqueness conflicts skip; .xcodeproj is never directly edited.
Relationship to other skills
| Skill | Layer | Asks |
|---|---|---|
pitfall-verification |
artifact | Will this work? |
quality-review |
artifact | Will this feel premium? |
macos-native-review |
artifact | Is this Apple-native? |
macos-e2e-scaffold |
project | Is this E2E-tested? |
swiftui-expert-skill (Antoine) |
code | Is the view code idiomatic? |
swift-testing-expert (Antoine) |
code | Is unit-test code idiomatic? |
swift-concurrency-expert (Antoine) |
code | Is async/await usage correct? |
core-data-expert (Antoine) |
code | Is the persistence layer well-designed? |
macos-e2e-scaffold is one of the two scaffold skills (with /ios-e2e-scaffold) that create test infrastructure rather than reviewing artefacts. This warrants the manual-only invocation pattern (no auto-trigger) — skill should run with full user awareness, not as a pipeline step.
Runner script template
scripts/run-uitests.sh:
#!/usr/bin/env bash
# Auto-generated by /macos-e2e-scaffold v2.53.0
# Runs UI tests on this machine or in a VM rig, and emits one Claude-readable JSON
# summary either way. Requires Xcode 16+ for the JSON xcresulttool format.
#
# Layering: this script CHOOSES the executor; `vm-e2e` OWNS the lease and the guest.
# Nothing here knows what a VM is beyond "a command on PATH that speaks this JSON".
set -uo pipefail
# Run from the project root regardless of where the caller stood. Every path below
# is relative to it — `.gstack/e2e-executor` above all — and read from a subdirectory
# the pin simply is not found, so an INVALID pin degrades to a silent host run: the
# one outcome the marker exists to prevent. (Borrowed from the rig session's runner,
# which had this right; the same class of bug Codex found in vm-hygiene.sh, fixed
# there and missed here.)
cd "$(cd "$(dirname "$0")/.." && pwd)" || { echo "cannot resolve project root" >&2; exit 2; }
SCHEME="<APP>"
UITEST_TARGET="<TARGET_DIR>" # <App>UITests, or <App>macOSUITests on multiplatform
RESULT_BUNDLE="$(mktemp -d)/uitests.xcresult"
# --- Executor pin ---------------------------------------------------------------
# host | vm; absence means host. Only those two strings are valid: a typo that
# silently ran on the host is the failure this pin exists to prevent, because the
# run still looks fine afterwards.
EXECUTOR=host
if [ -f .gstack/e2e-executor ]; then
# `$( )` strips exactly the trailing newline the generators write and nothing
# else, so the case below sees the whole file: `vm ` keeps its space, `v m` its
# gap, and a second line stays attached. Every one of those is then BLOCKED.
# `tr -d '[:space:]'` and `head -1 | sed` both normalise such junk into a legal
# value instead — the opposite of validating it (Codex, 2.53.0).
EXECUTOR=$(cat .gstack/e2e-executor)
case "$EXECUTOR" in
host|vm) ;;
*) echo "BLOCKED — invalid .gstack/e2e-executor: '${EXECUTOR}' (expected host or vm)" >&2
exit 2 ;;
esac
fi
# --- VM path --------------------------------------------------------------------
if [ "$EXECUTOR" = vm ]; then
if command -v vm-e2e >/dev/null 2>&1; then
VM_JSON="$(mktemp)"
vm-e2e "$PWD" "$SCHEME" --only-testing "$UITEST_TARGET" \
--json "$VM_JSON" --result-bundle "$RESULT_BUNDLE" >&2
VM_STATUS=$?
# A rig FAULT and a test FAILURE are different events. The rig produced a usable
# result iff the JSON parses and carries a total; anything else — empty stdout, a
# half-written file, an `error` field, a VM that never booted — is a fault. Do NOT
# fall back to the host here: that turns a real defect into a silently slower pass,
# and the whole reason to notice a rig fault is that it is a defect.
# Exit 2 from the rig means "could not run" — lease, boot, transfer, timeout, or
# zero tests executed. That is declared, not inferred, so it settles the question
# before any JSON is read (rig contract, virtual-mac dff0a26).
if [ "$VM_STATUS" -eq 2 ]; then
echo "E2E RIG FAILED: vm-e2e could not run (exit 2)." >&2
jq -r '.error // empty' "$VM_JSON" 2>/dev/null >&2 || true
echo "Not falling back to the host — a rig fault is the thing to fix, not to route around." >&2
rm -f "$VM_JSON"
exit 2
fi
# The JSON checks below stay as defence in depth, and as the only signal available
# from a rig older than the three-valued contract. A parseable object is not
# automatically a usable result: `{"total":1,"executed":1,"error":"copy failed"}`
# parses, and a missing `failed` defaults to 0 — so without these checks a rig that
# reported its own failure would print as a green summary.
# Require: no error field, and every count that EXISTS is a number. `// 0` would
# treat `"skipped": null` as absent, but null means unknown — executed would then
# read total-0, higher than reality, blinding the green-and-empty check.
if ! jq -e 'type == "object"
and (.error // null | . == null)
and (.total | type == "number")
and ((has("failed") | not) or (.failed | type == "number"))
and ((has("skipped") | not) or (.skipped | type == "number"))' "$VM_JSON" >/dev/null 2>&1; then
echo "E2E RIG FAILED: vm-e2e produced no usable result (exit ${VM_STATUS})." >&2
jq -r '.error // empty' "$VM_JSON" 2>/dev/null >&2 || true
echo "Not falling back to the host — a rig fault is the thing to fix, not to route around." >&2
rm -f "$VM_JSON"
exit 2
fi
# ALWAYS derive `executed` from the two counts just validated — never trust the
# rig's own field. A present-but-non-numeric `.executed` would pass the guard above
# (which checks total/failed/skipped) and then make `[ -eq 0 ]` fail with status 2;
# without `set -e` the script carries on and exits 0. Deriving removes the field
# from the trust surface entirely (Codex, 2.53.0).
SKIPPED=$(jq -r '.skipped // 0' "$VM_JSON")
FAILED=$(jq -r '.failed // 0' "$VM_JSON")
EXECUTED=$(( $(jq -r '.total' "$VM_JSON") - SKIPPED ))
jq --argjson ex "$EXECUTED" '. + {executor: "vm", executed: $ex}' "$VM_JSON"
rm -f "$VM_JSON"
echo "executor=vm skipped=${SKIPPED} executed=${EXECUTED}" >&2
[ "$EXECUTED" -eq 0 ] && { echo "FAILED: 0 tests executed — green and empty is not a pass." >&2; exit 1; }
[ "$FAILED" -ne 0 ] && exit 1
# The counts can look clean while the rig still failed — an xcodebuild
# infrastructure error, a transfer that half-completed. The rig's own exit code
# carries that, so never discard it just because the summary parsed (Codex, 2.53.0).
if [ "$VM_STATUS" -ne 0 ]; then
echo "vm-e2e exited ${VM_STATUS} despite a clean-looking summary — treating as failure." >&2
exit "$VM_STATUS"
fi
exit 0
fi
# Rig absent is NOT a rig fault: a committed `vm` pin must not brick the repo on
# every Mac without the rig. Run on the host, but never silently.
echo "executor=vm requested, rig not found on this host — running on host without lease" >&2
# CI variables do not cover `claude --print` or a scheduled run, and `[ ! -t 1 ]` is
# useless here because an agent's Bash tool always pipes stdout — it would refuse
# every interactive run too. So the caller, which actually knows the session kind,
# says so via E2E_NONINTERACTIVE; e2e-route sets it. (Codex, 2.53.0)
if [ -n "${CI:-}${GITHUB_ACTIONS:-}${E2E_NONINTERACTIVE:-}" ]; then
echo "Refusing in a non-interactive session: nobody reads that warning, and an" >&2
echo "unleased host run can collide with another. Install the rig or pin host." >&2
exit 2
fi
EXECUTOR="vm→host-fallback"
fi
# --- Host path ------------------------------------------------------------------
# Unattended UI tests need automation permission; warn, do not stop — the run may
# still be watched by a human who can grant it.
automationmodetool 2>&1 | grep -c 'DOES NOT REQUIRE' >/dev/null 2>&1 || \
echo "note: could not confirm automation mode; a UI test may stall on a permission prompt" >&2
# Run the tests. Capture xcodebuild's OWN exit status via PIPESTATUS — piping into
# `tail` would otherwise mask a non-zero status (tail returns 0), making a failing
# committed-regression run look green in CI.
xcodebuild test \
-scheme "$SCHEME" \
-destination 'platform=macOS' \
-only-testing:"${UITEST_TARGET}" \
-resultBundlePath "$RESULT_BUNDLE" \
-quiet 2>&1 | tail -50
TEST_STATUS=${PIPESTATUS[0]}
# Parse xcresult to JSON summary (Xcode 16+); fall back to plaintext on older Xcode.
SUMMARY=$(xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE" --format json 2>/dev/null \
| jq --arg ex "$EXECUTOR" --arg rb "$RESULT_BUNDLE" \
'{total: .totalTestCount, passed: .passedTests, failed: .failedTests,
skipped: (.skippedTests // 0),
executed: (.totalTestCount - (.skippedTests // 0)),
executor: $ex, xcresult: $rb,
results: [((.testFailures // [])[]) | {test: .testIdentifier, file: .sourceCodeContext.location.filePath, line: .sourceCodeContext.location.lineNumber, message: .failureText}]}' 2>/dev/null)
# NOTE: `.testFailures` is null/absent on an all-green run; `(.testFailures // [])`
# guards `null | .[]` so passing runs still emit clean JSON instead of falling back.
if [ -n "$SUMMARY" ]; then
printf '%s\n' "$SUMMARY"
SKIPPED=$(printf '%s' "$SUMMARY" | jq -r '.skipped')
EXECUTED=$(printf '%s' "$SUMMARY" | jq -r '.executed')
echo "executor=${EXECUTOR} skipped=${SKIPPED} executed=${EXECUTED}" >&2
# "Green and empty" — everything skipped, nothing run, exit 0 — reads as success and
# is the most dangerous result this pipeline can produce. `executed` is the number
# that says whether anything actually happened.
if [ "$EXECUTED" -eq 0 ]; then
echo "FAILED: 0 tests executed — check fixtures and -only-testing." >&2
exit 1
fi
else
echo "(Xcode 16+ JSON format unavailable — falling back to plaintext)"
xcrun xcresulttool get --path "$RESULT_BUNDLE" 2>/dev/null | tail -100 || true
fi
# Exit with the REAL test status so CI / committed-regression runs fail when tests fail.
exit "$TEST_STATUS"