Incident Response

Incident Response — Full Lifecycle

Don't panic, don't guess, lower machine entropy.

What This Skill Is

A unified incident response skill covering the full lifecycle:

1. Triage — six-step playbook with structured logging, backoff/circuit-breaker, and verification-as-terminal-state
2. Escalation — severity classification and canonical escalation ladder
3. Resolution — postmortem, institutional memory, and prevention

When to Use

• A federation organ is red, unreachable, or failing health checks
• A VAULT999 entry, memory record, or constitutional artifact looks wrong
• A constitutional floor (F1–F13) trips or is suspected to be breached
• Arif reports a bug, anomaly, or service impact
• A runtime alert requires immediate structured response
• Service outages, security breaches, constitutional violations, or agent misbehavior

When NOT to Use

• Do not use for curiosity or "what if" probes. Confirmed or strongly suspected incidents only.
• Do not use to bypass the arifOS kernel for mutating, irreversible, or sovereign-class actions.
• Do not apply patches without containment, reversible staging, and kernel SEAL when required.
• If the root cause is upstream (cloud provider, OS, network), escalate instead of patching locally.

§1. INCIDENT SEVERITY

When in doubt, escalate one level higher.

§2. SIX-STEP PLAYBOOK

Step 0: Detect Restart Loop (Circuit Breaker)

Before doing anything else, check if the service is in a restart loop.

systemctl show <service> -p NRestarts 2>/dev/null

Circuit breaker rules:

• NRestarts > 5 in 5 minutes → STOP THE LOOP: systemctl stop <service>
• NRestarts > 20 in 1 hour → escalate to 888_HOLD
• Log the circuit breaker event
• Do NOT restart until root cause is found

Step 1: Sense

Establish observable facts before interpreting.

• Run organ health probes: systemctl status <unit> and journalctl -u <unit> --since '5m ago'
• Use arif_observe mode=vitals or mode=search for federation-wide signals
• Capture timestamps, error lines, and affected service names verbatim
• Log every probe: {who, what: "sense", why: <incident>, result: <findings>}

Step 2: Scope

Classify the incident to prevent scope creep.

Stop if scope starts expanding mid-diagnosis. Re-scope and re-authorize.

Step 3: Contain

Protect recoverability before changing anything.

• If data-loss risk exists: snapshot DB / vault / git state / config before any patch
• If no data-loss risk: document current state and defer containment
• For irreversible changes: route through arifOS kernel and obtain SEAL or sovereign ack
• Apply the minimum change that stops active damage

Step 4: Diagnose

Read, recall, and correlate. Stop hypothesizing when evidence explains the symptom.

• Read recent logs and config diffs
• Recall prior incidents and deployments with arif_memory mode=recall
• Check recent git commits, deploys, and dependency changes
• Check for patterns: same symptom in last 7 days? → partial-fix, not new incident
• Name the root cause with confidence level and supporting evidence

Step 5: Patch — One Change at a Time

Minimum reversible change, committed and verified.

• Draft the smallest fix that addresses the root cause
• One mutation per step. Never batch 3 fixes and restart
• Prefer commits over manual edits
• Deploy through the organ's standard path
• Verify after EVERY change — if the fix didn't work, revert and try next hypothesis
• Backoff strategy: 1st retry: 5s, 2nd: 30s, 3rd: 120s (escalate)
• If the patch is irreversible, apply 888 HOLD before continuing

Step 6: Postmortem + Structured Log

Close the loop with institutional memory.

• If a floor was breached, seal the postmortem to VAULT999 as witness
• Write postmortem to /root/INCIDENTS/<YYYY-MM-DD>-<slug>.md with:
  • Trigger, Scope classification, Root cause and evidence, Fix applied, Verification result, Prevention measures
• If the same symptom recurs within 7 days, treat as partial-fix pattern, not new incident
• Final structured log: {who, what: "postmortem", why: <incident>, result: {...}}

§3. ESCALATION LADDER

Agent detects incident
    ↓
Agent applies skill (if trained)
    ↓
Escalate to domain agent (GEOX/WEALTH/WELL/A-FORGE)
    ↓
Escalate to AAA control plane (routing + visibility)
    ↓
Escalate to arifOS 888_JUDGE (constitutional / irreversible)
    ↓
Escalate to Arif (human sovereign)

Notification Matrix

Investigation Skills

• Service down → vps-ops health probes
• Secret leaked → FORGE-secret-hygiene
• Agent misbehaving → re-read SOUL.md
• Constitutional violation → authority detection

§4. FIRE-TIME REAUTHORIZATION (WAJIB 5)

A decision made at write-time is NOT automatically valid at fire-time. Authorization must be re-judged.

Affected surfaces

Cron jobs, queued workers, dependency-update PRs, scheduled deployments, delayed shell jobs, retry queues, event-triggered automation, long-running MCP tasks.

The required invariant

Every deferred mutation must be judged TWICE: write-time + fire-time.

At fire time, re-check: identity/session validity, lease expiry, current branch/commit, current target state, changed blast radius, new evidence, human approval validity, dependency health, rollback availability, revocation status.

Failure behavior

A scheduled action with expired authority MUST become HOLD.

write_time_authorization.expiry < now() → return 888_HOLD at fire time.

§5. ORGAN DISAGREEMENT DOCTRINE (WAJIB 7)

When GEOX, WEALTH, and WELL recommend incompatible actions, the resolution order:

1. Hard veto conditions (any organ may trigger HOLD with evidence)
2. Blast-radius precedence (organ owning dominant irreversible consequence)
3. Pareto search (seek alternative satisfying all hard constraints)
4. F13 escalation (if no acceptable option → escalate to Arif)

Hard veto table

Forbidden Actions

• NEVER cover up an incident
• NEVER delay escalation to avoid "bothering" someone
• NEVER destroy logs or evidence
• NEVER restart a service without understanding why it failed
• NEVER patch a production organ without sensing, scoping, and containing first
• NEVER apply an irreversible patch without 888 HOLD / sovereign ack
• NEVER skip the postmortem for constitutional or repeated incidents
• NEVER suppress or omit Ω₀ (uncertainty) in incident receipts

Escalation Path