Cfn Ops

CFN Ops Skill (MegaPlan Operations-Design Phase)

Purpose: Design the operational surface of a feature BEFORE it ships, not after it breaks. Threat model, observability, rollout, success metrics, failure modes, capacity/cost, rollback, runbook — authored as design artifacts, not caught as review checkboxes. This is the "blind past merge" fix (PLANNING_PIPELINE_GAPS root cause 3): a plan must think through how the thing runs in production and how we will know it worked.

Phase: Operations-design. DAG level 6 in cfn-megaplan. Runs in parallel with cfn-design and cfn-test-plan.

Owns gaps: G10 (threat model), G11 (observability), G12 (rollout), G14 (failure-mode), G18 (success metrics), G23 (capacity/infra), G27 (rollback rehearsal), G28 (runbook), G31 (cost model).

When to Use

• Auto-invoked by cfn-megaplan at L6 for beta and enterprise tiers.
• mvp tier SKIPS this phase entirely. The orchestrator does not spawn cfn-ops at mvp (mvp profile sets ops to light with everything dropped, and proves correctness only — no ops surface). If you are reading this at mvp, stop: ops design is out of scope for a throwaway.
• Standalone for ops review of an existing feature heading to production.

Active for beta+ is gated by tier, not a build flag. It always runs at beta+ even for a pure-backend feature, because every feature has a production surface (logs, rollout, failure behavior, rollback).

Input

Required:

• planning/<slug>/SPEC_<slug>.md — success criteria, NFRs, audience.
• planning/<slug>/ARCH_<slug>.md — components, integration points, failure-mode inventory, cross-cutting (observability/auth hooks), data-flow diagram.

Conditional:

• planning/<slug>/DATA_<slug>.md — present only if the build has a db flag. Source for tables touched, RLS, PII, retention (feeds threat model + rollback + capacity).

Refuse to run if SPEC or ARCH is missing or still draft with unresolved [OPEN] items. Ops design built on an unstable arch is wasted.

Tier directives (what the orchestrator passes)

The orchestrator passes Tier, Directive, extras, drops. Resolve each protocol phase against the tier:

beta extras: observability, rollout, metrics, rollback_rehearsal. beta drops: threat_full, capacity_full, runbook_full (these run light, not off). enterprise extras: threat, observability, rollout, canary, metrics, failure_mode, rollback_rehearsal, capacity, cost_model, runbook (all full).

Security floor (never scales down, from megaplan floor): regardless of tier, if the feature serves HTTP, the rollout MUST confirm security headers (HSTS, CSP, X-Frame-Options) are applied via the shared middleware, not per-route. If the feature calls any LLM in a loop, capacity MUST name a --budget=<usd> cap. Anthropic API calls are banned in project code — if the design references an LLM provider, it must not be anthropic:* (per global CLAUDE.md replacement map). Flag any violation as [OPEN].

Protocol

Phase 1: Threat model (beta light / enterprise full STRIDE) — G10

Walk each data-flow edge from the ARCH data-flow diagram. For each, enumerate the relevant STRIDE categories and name the control that mitigates each. Security stops being a checklist and becomes a per-edge design.

STRIDE = Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege.

Output table (one row per threat, grouped by data-flow):

| Data flow                  | STRIDE | Threat                              | Control                                         | Residual |
|----------------------------|--------|-------------------------------------|-------------------------------------------------|----------|
| Client -> POST /api/orders | S      | Forged caller identity              | JWT verify in shared auth middleware            | low      |
| Client -> POST /api/orders | T      | Tampered order total in transit     | TLS + server recomputes total from line items   | low      |
| Client -> POST /api/orders | I      | Order leaks across tenants          | RLS policy tenant_id = auth.tenant (DATA phase)  | low      |
| Client -> POST /api/orders | D      | Flood of order writes               | Per-tenant rate limit 100/min (ARCH §6)         | med      |
| Worker -> Postgres         | R      | No audit of who changed status      | append-only status_events row per transition    | low      |

beta floor: EVERY externally-reachable data-flow edge gets >=1 STRIDE row, and S, T, and I are each evaluated per edge at minimum. A category skipped on an edge gets an explicit "n/a: " cell; the reason is required, never blank. enterprise: every edge, all 6 categories, plus residual risk and an owner for any med/high residual.

Cross-reference: HTTP edges must confirm the security-headers middleware (HSTS/CSP/X-Frame-Options). DB edges must reference the RLS policy from DATA_<slug>.md. Never hand-roll auth/crypto/token parsing here — point at the vetted middleware.

Phase 2: Observability design (beta+) — G11

Design what gets emitted at every decision point so an on-call engineer can answer "what happened" without redeploying. Logging is designed here, not added later.

Log lines at decision points. For each branch where the system makes a consequential choice (auth allow/deny, gate pass/fail, retry, abort, fallback fired, payment captured), specify the exact structured log line. Every line carries correlation IDs: request_id, the primary entity_id, and tenant_id where multi-tenant.

Log-line spec format (each line carries OBS-id, criticality, and verify):

OBS-id: OBS-1
event: auth.deny
level: warn
criticality: alert
verify: required
when: JWT verify fails OR tenant mismatch on POST /api/orders
fields: { request_id, user_id, tenant_id, reason: "expired"|"tenant_mismatch"|"missing", route }
PII: user_id only (no email/token in line) — redact token as [REDACTED]

Enumerate at minimum: every allow/deny, every gate pass/fail, every retry+abort, every fallback/degradation trigger, every external-call failure.

Metrics. Name each metric, its type (counter/gauge/histogram), labels, the SLO it serves, and its OBS-id, criticality, and verify:

| OBS-id | Metric                      | Type      | Labels            | SLO / use                        | Criticality (alert|slo|core-signal|diagnostic) | Verify (required | exempt: <reason>) |
|--------|-----------------------------|-----------|-------------------|----------------------------------|-----------------------------------------------|--------------------------------------|
| OBS-2  | orders_created_total        | counter   | tenant, status    | success-rate KPI (Phase 4)       | slo                                           | required                             |
| OBS-3  | order_write_latency_seconds | histogram | route             | p95 < 300ms                      | slo                                           | required                             |
| OBS-4  | order_dep_down_total        | counter   | dep               | alert when > 0 over 5m           | alert                                         | required                             |
| OBS-5  | order_debug_trace           | counter   | route             | dev diagnostics only             | diagnostic                                    | exempt: debug log, owes no test      |

Criticality rule (verbatim): verify-required iff the signal backs an alert threshold or on-call query, defines a Phase 4 KPI/guardrail, or is the runtime-observed signal of a [core] FR. Debug logs never owe tests.

Ops names signals (OBS-n), never AC ids: cfn-test-plan Phase 3 maps each verify: required OBS-n to an AC that asserts the signal fires for the test's own input. OBS-id is the greppable token and the key behind Bar A counters obs_required_total/obs_required_mapped.

Traces. Name the span boundaries for the primary flow (entry span → external-call child spans) so a slow request is attributable to a stage.

The on-call query. Write the EXACT query an on-call would run for the most likely incident. Not "check the logs" — the literal query:

# "orders failing for tenant X in the last 15m"
log query: event="order.write.fail" AND tenant_id="X" AND ts > now-15m
          | stats count by reason

This query is part of the deliverable, not a suggestion.

Phase 3: Rollout (beta flags / enterprise canary) — G12

No feature reaches 100% of traffic on deploy. Design the staged path.

• Feature flag: name, default (off), kill-switch owner, where evaluated (server-side). The flag wraps the new code path; old path stays live behind it.
• Staged percentages: e.g. internal → 1% → 10% → 50% → 100%, with the dwell time and the go/no-go check at each stage.
• Canary (enterprise): wire cfn-canary — it polls the deploy URL for 10 minutes post-ship checking errors, latency regression, availability. State the canary pass criteria as binary checks:

| Stage | % traffic | Dwell | Promote when (binary)                              |
|-------|-----------|-------|----------------------------------------------------|
| canary| 1%        | 10m   | error_rate < 0.5% AND p95 within 10% of baseline   |
| ramp  | 10%       | 1h    | order_dep_down_total == 0 AND success KPI stable   |
| full  | 100%      | —     | ramp green, no rollback triggered                  |

Security floor: confirm the new route serves the shared security-headers middleware BEFORE the canary opens (a flagged route that bypasses middleware is a regression).

Phase 4: Success metrics / KPIs (beta+) — G18

Tests-pass is not feature-works. Define how we know in PRODUCTION that the feature did its job. Acceptance-in-production, expressed as a measurable query, not prose.

| KPI                         | Definition (query)                                  | Target          | Window |
|-----------------------------|-----------------------------------------------------|-----------------|--------|
| Order completion rate       | orders_created_total{status=ok} / attempts          | >= 95%          | 7d     |
| Median time-to-confirm      | histogram_quantile(0.5, order_confirm_latency)      | < 2s            | 24h    |
| Guardrail: error budget     | 5xx on /api/orders / total                          | < 1% (else roll)| 1h     |

Each KPI maps back to a SPEC success criterion. If a success criterion has no measurable prod KPI, flag it [OPEN] — we would ship blind to whether it worked. enterprise adds per-segment KPIs and explicit guardrail metrics that trigger rollback.

Phase 5: Failure-mode / degradation (beta light / enterprise FMEA) — G14

For each dependency the feature relies on (from ARCH external integrations + failure-mode inventory), design what happens when it is down or slow. The system must degrade, not collapse.

beta light — name behavior per critical dep:

| Dependency   | Down behavior                         | Timeout budget | Fallback                    |
|--------------|---------------------------------------|----------------|-----------------------------|
| Postgres     | reject writes, 503 + Retry-After      | 2s connect/5s  | none (hard dep) — queue?    |
| Payment API  | mark order pending, async retry       | 3s             | retry queue, notify later   |
| Redis cache  | bypass cache, read DB                 | 200ms          | DB read (slower, correct)   |

enterprise full FMEA — add columns: Severity (1-5), Likelihood (1-5), Detection (how the observability from Phase 2 catches it), RPN = S×L×(6-D), and a circuit-breaker spec (threshold, open duration, half-open probe) for each external call. Order rows by RPN; anything high gets a designed mitigation, not a TODO.

Timeout budgets must sum to less than the request's overall deadline — state the budget so cascading timeouts can't blow the SLA.

Phase 6: Rollback rehearsal (beta+ tested) — G27, G50

"Revert the migration" is a wish, not a procedure. Write the actual undo steps AND the evidence they were exercised (dry-run in staging, or a documented reasoning trace if no staging).

Executable rollback evidence (db flag + reversible migration). When the db build flag is set AND DATA_<slug>.md §5 declares a reversible migration, the rehearsal evidence is not reasoning: it is the cfn-migration-rehearsal invocation, cited verbatim from DATA §5's line:

CFN_SCRATCH_DATABASE_URL=<scratch> ./.claude/skills/cfn-migration-rehearsal/execute.sh --up <NNNN.up.sql> --down <NNNN.down.sql>

• Reasoning-only evidence is acceptable ONLY when no scratch DB is possible; state why and emit a WARN row so the gap is visible, never a silent downgrade.
• An irreversible migration carries n/a: <reason from DATA §5> (the same one-line reason DATA §5 recorded), not a rehearsal command.
• The rehearsal never runs against DATABASE_URL — the skill already refuses any prod look-alike and requires CFN_SCRATCH_DATABASE_URL. This is design-side only; the loop executes it.

This is the ops-side meaning of the existing rollback_rehearsal profile token (G50) — no new ops token is added; the token's meaning is extended here. Phase 2's OBS work is likewise the extended meaning of the existing observability token (G49).

Trigger: guardrail error budget breached (Phase 4) OR canary fails (Phase 3).
Steps (tested):
  1. Flip feature flag `orders_v2` -> off (instant; no deploy). Verifies: old path serves, new code dormant.
  2. If schema changed: run down-migration NNNN_orders_v2_down.sql. Pre-checked: down migration is reversible, drops only new columns, touches no existing rows (WHERE-scoped, no unscoped DELETE).
  3. Confirm: orders_created_total resumes on old path; error budget recovers within 5m.
Data safety: down-migration MUST NOT drop columns with data the old path needs. If new rows were written under the new schema, state the reconciliation step.
Rehearsal evidence: dry-run in staging on <date> / OR reasoning trace why each step is safe.

Floor: any rollback that touches the DB inherits the test-database safety rules — scoped operations only, no unscoped DELETE/TRUNCATE, trace the FK cascade. A rollback that wipes data is worse than the bug.

Phase 7: Capacity / infra / cost (enterprise; beta light) — G23, G31

enterprise — full:

• Topology: where it runs (which app/worker), what it talks to, scaling unit (per-request, per-worker).
• Scaling: expected RPS, the bottleneck resource (DB connections, CPU, memory), autoscale trigger and ceiling. Resource budgets are named constants in shared config, not magic numbers.
• Cost-per-call: infra cost + LLM token cost if the path calls a model. State the model, tokens-per-call estimate, and price (from ~/.claude/model-pricing.md). Anthropic providers are banned — if an LLM is in the loop it routes to the allowed provider (e.g. xai:grok-4-1-fast-non-reasoning), never anthropic:*.
• Budget cap: any long-running or batch pipeline that bills API MUST declare --budget=<usd> and refuse to start without it (cost-safety floor). State the cap.

| Resource          | Budget / cap                  | Bottleneck at       | Cost/call            |
|-------------------|-------------------------------|---------------------|----------------------|
| DB pool           | 20 conns (shared config const)| ~400 RPS            | —                    |
| LLM summarize     | --budget=5.00 per batch       | provider rate limit | ~$0.002 (grok-4-1)   |

beta floor — name at least one budget row whose value is a named constant in shared config (cite the constant name, not a magic number), plus the --budget=<usd> row whenever an LLM is in the loop. A beta Phase 7 with zero budget rows, or an LLM in the loop with no --budget row, is incomplete. Skip full topology.

Phase 8: Runbook / on-call (beta light / enterprise full) — G28

How a human operates this feature after merge.

beta light — symptom → first action table:

| Symptom                          | First action                                      |
|----------------------------------|---------------------------------------------------|
| Orders failing, error budget red | Flip flag orders_v2 off (Phase 6 step 1)          |
| Latency p95 spiking              | Check order_write_latency by route; check DB pool |
| Payment dep down alert           | Confirm retry queue draining; no manual replay    |

enterprise — full on-call doc: each alert → owner, escalation path, the Phase 2 query to confirm, the Phase 6 rollback trigger, and a "do NOT do" list (e.g. do not replay payment retries manually). Reference the relevant cfn-canary run and dashboards.

Output

Artifact location. Every artifact of one plan lives in that plan's own directory, planning/<slug>/. Under /cfn-megaplan, /cfn-megaplan-lite, or /cfn-spa-plan the orchestrator hands you the exact path plus a Plan dir: line — write there, and read the input paths it gives you verbatim. Invoked standalone, read with $HOME/.claude/skills/cfn-megaplan/lib/plan-paths.sh resolve <slug> <basename> (per-plan dir first, legacy flat planning/ second) and write to planning/<slug>/. Never split one plan across two locations.

Write to: planning/<slug>/OPS_<slug>.md

Template (include only the phases active for the tier; mark skipped phases N/A (tier)):

# Operations Design: <task>

**Date:** <YYYY-MM-DD>
**Tier:** beta | enterprise
**Spec:** planning/<slug>/SPEC_<slug>.md
**Arch:** planning/<slug>/ARCH_<slug>.md
**Data:** planning/<slug>/DATA_<slug>.md (if db)
**Status:** draft | reviewed | locked

## 1. Threat Model (STRIDE)
| Data flow | STRIDE | Threat | Control | Residual |

## 2. Observability
### Log lines (decision points)
<event specs; each carries OBS-id, criticality (alert|slo|core-signal|diagnostic), verify (required | exempt: <reason>)>
### Metrics
| OBS-id | Metric | Type | Labels | SLO/use | Criticality (alert|slo|core-signal|diagnostic) | Verify (required | exempt: <reason>) |
(verify-required iff the signal backs an alert/on-call query, defines a Phase 4 KPI/guardrail, or is the runtime-observed signal of a [core] FR; debug logs never owe tests; OBS-id is the greppable token cfn-test-plan consumes and the Bar A `obs_required_total/obs_required_mapped` key)
### Traces
<span boundaries>
### On-call query
<literal query>

## 3. Rollout
- Feature flag: name, default, kill-switch owner
| Stage | % | Dwell | Promote when (binary) |
- Security headers confirmed via shared middleware: yes/no
- cfn-canary wired (enterprise): yes/no

## 4. Success Metrics / KPIs
| KPI | Definition (query) | Target | Window |

## 5. Failure Modes
| Dependency | Down behavior | Timeout budget | Fallback | (S|L|D|RPN if FMEA) |

## 6. Rollback Rehearsal
- Trigger, tested steps, data safety, rehearsal evidence
- db + reversible: cfn-migration-rehearsal invocation cited verbatim from DATA §5 (executable evidence); no scratch DB -> WARN row + why; irreversible -> `n/a: <reason from DATA §5>`

## 7. Capacity / Cost
| Resource | Budget/cap | Bottleneck | Cost/call |
- --budget cap (if LLM in loop):

## 8. Runbook
| Symptom | First action |

## [OPEN]
<decisions needing the user: unmeasurable success criterion, banned provider, missing budget cap, header bypass>

Handoff

OPS_<slug>.md is consumed at L7 by /write-plan, which folds rollout/observability/rollback into the implementation roadmap (e.g. the flag wrapper, the log lines, the down-migration become concrete build tasks via its "Ops Integration Tasks" section). Bar A (verifiable-done) turns each KPI and canary criterion into an executable AC. cfn-plan-review (L8) blast-radius-checks the ops surface.

Return (to orchestrator)

Return exactly:

• Artifact path: planning/<slug>/OPS_<slug>.md
• A 3-line summary (STRIDE edges covered with row count, flag name + rollout stages, rollback trigger + down-migration named).
• Floors line: STRIDE floor met yes/no (every external edge has >=1 row, S/T/I evaluated), budget floor met yes/no (>=1 named-constant row; --budget row present if LLM in loop).
• Any [OPEN] items needing a user decision (unmeasurable success criterion, banned provider, missing budget cap, header bypass).

Anti-Patterns

• "We'll add monitoring later." Later means after the incident, blind. Log lines at decision points are designed here or the on-call has nothing.
• "Rollback = just redeploy / revert the migration." Untested undo is a wish. Phase 6 requires the actual steps and evidence they work.
• "No success metric." If you can't state the prod query that proves it worked, you ship blind and never learn. Every KPI maps to a SPEC criterion.
• Security headers per-route instead of shared middleware. HSTS/CSP/X-Frame-Options bypassed on the new route is a regression the rollout must catch BEFORE canary.
• Threat model as a checkbox ("security reviewed: yes") instead of a per-data-flow STRIDE table with a named control each.
• LLM in the loop with no --budget cap, or a path routing to a banned anthropic:* provider.
• Timeout budgets that sum past the request deadline — cascading timeouts blow the SLA you set in Phase 4.
• Unscoped DELETE/TRUNCATE in a rollback — a rollback that wipes data is worse than the bug it undoes.
• Running ops at mvp. mvp proves correctness only; the orchestrator does not spawn this phase there.

Related

• Orchestrator: cfn-megaplan (L6, parallel with cfn-design, cfn-test-plan)
• Upstream inputs: cfn-spec, cfn-arch, cfn-data
• Wires: cfn-canary (post-deploy health monitoring, Phase 3)
• Specialists: security-specialist / technical-advisor (STRIDE, FMEA), devops-engineer / fly-io-specialist (capacity)
• Gates downstream: Bar A turns KPIs/canary criteria into executable ACs; cfn-plan-review blast-radius
• Backlog + rationale: docs/PLANNING_PIPELINE_GAPS.md (gaps G10, G11, G12, G14, G18, G23, G27, G28, G31)