AI Launch Check

AI Feature Launch Check

Iron Law: Do not output GO if any checklist item is ❌. A written exception in the Decision Record with an owner and deadline is required for every ⚠️ item. Verbal assertions do not count as evidence for any item.

Run the unified pre-production launch checklist from the AI Playbook (Layer 4 §4.1). This consolidates the pre-flight questions, pre-production gate, kill-switch tests, and adversarial test catalog into one launch review.

Procedure

1. Ask the user for the feature slug. Look for the Decision Record at docs/ai-decisions/<feature-slug>.md.
   • If it does not exist: stop. Tell the user to run /ai-decision-record first. Do not proceed without one.
2. Read the Decision Record. Cross-check every item below against what the record says.
3. For each checklist item below, ask the user to show evidence, not assert compliance. Examples of evidence:
   • "Output Contract implemented" → point to the validation code; read it.
   • "Eval set" → point to the file; show the size and a sample.
   • "Kill switch tested" → show the test run log.
4. Mark each item: ✅ verified / ⚠️ partial / ❌ missing.
5. Do not pass the launch check if any item is ❌. Partial is allowed only when the user provides a written exception in the Decision Record with an owner and a deadline.
6. Produce a launch decision: GO / NO-GO / GO-WITH-EXCEPTIONS with a written rationale.

The pre-flight (10 questions)

Walk through these first. Any "no" or "I don't know" is a NO-GO.

• 1. Can deterministic code solve this reliably? (If yes → why are we shipping AI?)
• 2. Is the input genuinely unstructured or ambiguous?
• 3. Does AI measurably beat code on accuracy, speed, cost, or UX?
• 4. Is the decision reversible, or is there a downstream verifier?
• 5. Off the critical latency path, or fast fallback exists?
• 6. What's the cost when AI is wrong, and who absorbs it?
• 7. Is human approval required for high-impact outcomes?
• 8. Can every AI output be audited and explained later?
• 9. Enough labeled examples for risk-tiered evaluation?
• 10. Deterministic fallback when AI fails or times out?

The artifacts checklist

Each item must point to a file, a log, or a running piece of code. Verbal assertions do not count.

• Decision Record signed off (docs/ai-decisions/<feature-slug>.md)
• Risk-tiered eval set with target metrics (size matches Layer 2 §2.14 for the declared risk tier)
• Output Contract implemented: schema → types → business rules → permissions → confidence → audit → route (Layer 2 §2.4)
• Adversarial tests in CI (Layer 3 §3.2): prompt injection, tool misuse, retrieval poisoning, bad schema outputs
• RAG requirements met if applicable (Layer 2 §2.7): hybrid search, metadata filtering, ACLs at query time, citations, no-answer test, separate retrieval/answer grading, chunk provenance
• Multi-modal handling reviewed if applicable (Layer 2 §2.8): each modality validated, cost recalculated, sandboxes for code execution
• Model and routing strategy decided (Layer 2 §2.9)
• Cost & latency budget filled (Layer 3 §3.4): max calls, max tokens, p50/p95/p99, cost ceiling, fallback after timeout, cache strategy, rate limit
• 12-month AI tax estimated and approved (Layer 2 §2.11)
• Vendor abstraction in place; model and prompt versions pinned (Layer 2 §2.12)
• Observability schema implemented (Layer 3 §3.5): model_version, prompt_version, input_hash, validation result, confidence, routed_to, downstream outcome
• Human review queue defined: what goes, who reviews, SLA per Layer 2 §2.1
• Data governance review completed (Layer 2 §2.13): context contents, logging, retention, training-use, tenant isolation, deletion propagation, redaction
• Security review completed (Layer 2 §2.5): the two questions answered with code, not prompts
• Rollout plan with promotion criteria (Layer 4 §4.2): offline → dogfood → shadow → beta → percentage → full
• Incident response plan (Layer 4 §4.3): named on-call, rollback triggers with thresholds, affected-user identification, quarantine, reversal, post-mortem template, leading indicators
• Kill-switch tests passing (Layer 4 §4.4): all eight tests in staging or shadow, with logs

Kill-switch tests — must all pass

Demand the test logs. If they don't exist, fail the launch check.

• Disable feature via flag → graceful degradation
• Force model downgrade → output contract still passes or routes to review
• Simulate vendor outage → timeout fires, fallback serves, no infinite retry
• Simulate retrieval poisoning → contract/security rejects; provenance logs source
• Simulate cost spike (10× tokens) → rate limit + cost ceiling fire; alerts route correctly
• Simulate prompt injection from user input AND from a retrieved doc → least-privilege scope prevents harm
• Roll back one prompt or model version → traffic shifts cleanly; metrics return to baseline
• Verify deletion propagation → user-deleted data gone from embeddings, caches, logs, evals within SLA

Decision output

After running through every item, output:

# Launch Check: <Feature Name>

**Decision**: GO | NO-GO | GO-WITH-EXCEPTIONS
**Reviewer**: <Claude Code session> — <date>
**Decision Record**: docs/ai-decisions/<feature-slug>.md

## Verified (✅)
- ...

## Partial — exceptions filed (⚠️)
- <item>: exception in Decision Record, owner <name>, deadline <date>

## Missing — blockers (❌)
- ...

## Recommendation
<one paragraph: ship, scope down, or stop>

## If GO-WITH-EXCEPTIONS: follow-up SLAs
- ...

Default stance

• Bias toward NO-GO. A launch check is a forcing function; if the team passes too easily, the check isn't doing its job.
• Do not let "we'll fix it after launch" pass for kill-switch tests, output contract, or security review. Those are pre-launch by definition.
• Tier-3+ features with no human approval gate are an automatic NO-GO unless the Decision Record has a written domain-risk exception.

If this skill cannot proceed

Abort condition: Decision Record missing
→ Hard stop. Output: "Run /ai-decision-record first. The launch check
  cross-references the Decision Record for every checklist item. Cannot proceed without it."

Abort condition: user provides verbal assertions instead of file:line evidence
→ Mark item ⚠️ at best, ❌ if critical (kill switch, output contract, security review).
→ Output: "Verbal assertions are not evidence. Point to the file, log, or test output."

Abort condition: all 8 kill-switch tests have no logs
→ Automatic NO-GO regardless of other items.
→ Output: "Kill-switch tests must be run in staging or shadow with logs. 'We tested it
  manually once' does not satisfy this requirement."

Verify (after producing the launch decision)

✅ Every checklist item has a status: ✅ / ⚠️ / ❌ — no blanks
✅ Every ⚠️ item has a named owner and a written deadline in the Decision Record
✅ Decision (GO / NO-GO / GO-WITH-EXCEPTIONS) matches the item statuses — no ❌ items in a GO
✅ Launch check report written to docs/ai-audits/<feature-slug>/04-launch-check.md

References

• AI Playbook Layer 4 §4.1 Unified launch checklist: .claude/skills/ai-playbook/playbook.md
• AI Playbook Layer 4 §4.2 Rollout strategy: .claude/skills/ai-playbook/playbook.md
• AI Playbook Layer 4 §4.4 Kill-switch tests: .claude/skills/ai-playbook/playbook.md
• AI Playbook Layer 2 §2.14 Eval size floors: .claude/skills/ai-playbook/playbook.md
• Prerequisite: /ai-decision-record (must exist before launch check runs)