Kelly App Skill Creator Tests

Kelly App Skill Creator Tests

Verify generated Kelly app skills without taking ownership of their product or runtime design. Treat the app source and creator contracts as inputs; never weaken either contract merely to make a test pass.

Ownership Boundary

• $kelly-app-skill-creator owns product design and creates the canonical <skill-root>/app/ project.
• $busabase-app-creator owns AirApp runtime, SDK, security, sync, and deployment constraints.
• This skill owns shared test infrastructure, per-skill fixtures and assertions, OSS/Cloud execution gates, failure artifacts, CI wiring, and test reporting.
• Keep pure domain and module tests in <skill-root>/app/test/. Keep cross-skill contract, process, browser, Busabase, OAuth, and AirApp tests under the repository-level tests/app-skills/ tree.

Required Reading

Before changing tests:

1. Read the target skill's SKILL.md, canonical app/, resource map, checks, and existing tests.
2. Read $kelly-app-skill-creator and the runtime rules it delegates to $busabase-app-creator.
3. Read references/test-matrix.md completely for suite boundaries, environment variables, and required assertions.

If a dependency skill is unavailable, inspect its checked-in source when present. Do not invent a weaker replacement contract.

Workflow

1. Discover canonical apps through skills/*/app/package.json. Do not fail legacy app/ directories that have not yet adopted the canonical contract.
2. Reuse one shared harness for process lifecycle, free ports, readiness, temporary directories, browser diagnostics, and Busabase startup.
3. Add a small target directory at tests/app-skills/<skill>/ containing only that skill's contract assertions, fixtures, routes, resources, and one representative workflow.
4. Run the app's own frozen install and deterministic check before cross-skill tests.
5. Run contract and local-server tests, then headless Chromium acceptance at 1280x820, 390x844, and 360x740.
6. Run the OSS suite against a pinned busabase server on a random loopback port and temporary data directory. Verify lazy provisioning, exact resources, idempotency, representative data, app restart, and Busabase restart.
7. Run Cloud OAuth as a separate suite only when every required credential variable is non-empty. Print an explicit skip with missing variable names otherwise. Never describe a skipped Cloud suite as validated.
8. Before any live Cloud suite, run deterministic fake-upstream connection conformance for zero, one, and multiple Spaces, invalid/stale selection, inbound-header bypass, transient auth failure, server change, and logout. These cases run on every PR and cannot be skipped because a Cloud account has only one Space.
9. Permit Cloud mutations only when the dedicated test Space and explicit mutation opt-in are both present. OAuth-only mode must not provision, seed, merge, or delete Cloud data.
10. When AirApp credentials and a disposable target are available, run deployed ambient-session and local-source parity acceptance separately from standalone Cloud OAuth.
11. Report every required suite as pass, fail, or skip, including the exact command and failure artifact location. Preserve a failing test when it reveals an app, SDK, Busabase, OAuth, or creator-contract defect.

Integrity Rules

• Pin Busabase, SDK, browser tooling, and package-manager versions used by CI.
• Bind test services to loopback, use random ports, and isolate every OSS run in a temporary data and credential directory.
• Use real Busabase APIs for integration tests. Mocks may test error branches but cannot satisfy OSS, Cloud, or AirApp acceptance.
• Keep provider-network tests separate from deterministic app tests. Assert response contracts and provenance, not volatile prices or counts.
• Never log passwords, API keys, OAuth codes, access/refresh tokens, Vault values, cookies, or unredacted response bodies that may contain them.
• Store Cloud OAuth credentials under a temporary HOME, revoke the session at teardown, and verify the browser did not receive token material.
• Do not run Cloud tests against an unspecified or personal Space. Require a dedicated test account; require a dedicated Space before any mutation.
• Capture screenshots and process/API diagnostics on failure, but redact secret query parameters, headers, cookies, and bodies first.
• Treat cleanup failure as a test failure when it can leave remote credentials or mutable test state behind.

Completion

Finish only when the target app's own checks, contract tests, local smoke, responsive Demo flow, and OSS Busabase suite pass. When Cloud variables exist, Cloud OAuth must pass; when they do not, report the explicit skip and remaining risk. Do not claim full creator conformance until required AirApp acceptance has also passed or has been explicitly scoped out by the user.