# FORGE-incident-triage

> Six-step incident response playbook with structured logging, backoff/circuit-breaker for restart loops, and verification-as-terminal-state. Lower machine entropy.

- Skill: `ariffazil/forge-incident-triage` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add ariffazil/forge-incident-triage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ariffazil/forge-incident-triage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ariffazil (https://skillmd.com/u/ariffazil)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/ariffazil/forge-incident-triage

---


# Incident Triage — Lower Entropy Response

The "don't panic, don't guess, lower machine entropy" playbook for federation incidents.

## Overview

This skill provides a disciplined six-step response for incidents affecting federation organs, constitutional floors, or sovereign-reported faults. It forces sensing and scoping before action, containment before diagnosis, and verification before closure. Every step leaves evidence suitable for VAULT999 witness.

### Core Principles
1. **Structured logs**: every action emits `{who, what, why, result}` — readable by humans and machines
2. **Backoff + circuit breaker**: stop noisy retries before they become noise (headscale style)
3. **Verification is terminal**: never stop at "I changed it" — only at "it's fixed and confirmed"
4. **One owner per incident**: one agent handles the entire triage; no parallel patches
5. **One change at a time**: record every mutation, verify before the next

## arifOS-ACT Embedding

Before using this skill on any mutating, irreversible, or high-blast-radius task:
1. **ART** — Attune (what is the real task?), Recognize (what class of power?), Test (fit · authority · evidence · blast · reversible).
2. **Kernel** — Route to arifOS for F1–F13 judgment if action class is Maker/Messenger/Mutator/Destroyer/Sovereign.
3. **ACT** — Apply narrow, Constrain scope, Trace witness, STOP before corruption.
4. **Receipt** — Leave evidence of what changed, why, and under whose authority.

## When to Use

- A federation organ is red, unreachable, or failing health checks.
- A VAULT999 entry, memory record, or constitutional artifact looks wrong.
- A constitutional floor (F1–F13) trips or is suspected to be breached.
- Arif reports a bug, anomaly, or service impact.
- A runtime alert requires immediate structured response.

## When NOT to Use

- **Do not use for curiosity or "what if" probes.** This skill is for confirmed or strongly suspected incidents only.
- **Do not use to bypass the arifOS kernel** for mutating, irreversible, or sovereign-class actions.
- **Do not apply patches** without containment, reversible staging, and kernel SEAL when required.
- If the root cause is upstream (cloud provider, OS, network), escalate instead of patching locally.

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| incident_signal | yes | What failed, tripped, or was reported (service name, floor, symptom) |
| affected_organ | yes | One or more federation organs involved |
| blast_radius | yes | organ-only / federation-wide / constitutional / sovereign |
| evidence_so_far | no | Logs, screenshots, alerts, or memory references already available |
| reversibility | yes | Can the suspected change be undone safely? |

## Procedure

### Step 0: Detect Restart Loop (Circuit Breaker)

Before doing anything else, check if the service is in a restart loop.

```bash
# Check restart count
systemctl show <service> -p NRestarts 2>/dev/null
# If >5 in 5 minutes: STOP. Circuit breaker open.
```

**Circuit breaker rules:**
- If `NRestarts > 5` in 5 minutes → **STOP THE LOOP** immediately: `systemctl stop <service>`
- If `NRestarts > 20` in 1 hour → escalate to 888_HOLD — the service is masking a serious failure
- Log the circuit breaker event: `{who, what: "circuit breaker opened", why: "restart loop detected: NRestarts=<N>", result: "service stopped"}`
- Do NOT restart until root cause is found

### Step 1: Sense

Establish observable facts before interpreting.

- Run organ health probes: `systemctl status <unit>` and `journalctl -u <unit> --since '5m ago'`.
- Use `mcp__arifos__arif_observe` mode=vitals or mode=search for federation-wide signals.
- Use `mcp__arifos__arif_measure` mode=health for thermodynamic and resource state.
- Capture timestamps, error lines, and affected service names verbatim.
- **Log every probe**: `{who: <agent>, what: "sense", why: <incident>, result: <findings>}`

### Step 2: Scope

Classify the incident to prevent scope creep.

| Scope | Definition | Response |
|-------|------------|----------|
| Organ-only | One service or repo affected | Local containment + organ owner |
| Federation-wide | Multiple organs or A2A/MCP transport impacted | Federation ops + kernel notice |
| Constitutional | F-floor tripped or governance invariant violated | arifOS 888_JUDGE + witness |
| Sovereign | Human authority, safety, or dignity at risk | 888 HOLD + Arif |

Stop if scope starts expanding mid-diagnosis. Re-scope and re-authorize.

### Step 3: Contain

Protect recoverability before changing anything.

- If data-loss risk exists: snapshot DB / vault / git state / config before any patch.
- If no data-loss risk: document current state and defer containment.
- For irreversible changes: route through arifOS kernel and obtain SEAL or sovereign ack.
- Apply the minimum change that stops active damage (e.g., restart, scale, toggle feature flag).

### Step 4: Diagnose

Read, recall, and correlate. Stop hypothesizing when evidence explains the symptom.

- Read recent logs and config diffs with `Read` and `Grep`.
- Recall prior incidents and deployments with `mcp__arifos__arif_memory` mode=recall.
- Check recent git commits, deploys, and dependency changes.
- **Check for patterns**: same symptom in last 7 days? If yes → partial-fix, not new incident.
- Name the root cause with confidence level and supporting evidence.
- **Log diagnosis**: `{who: <agent>, what: "diagnose", why: <incident>, result: <root cause>}`

### Step 5: Patch — One Change at a Time

Minimum reversible change, committed and verified.

- Draft the smallest fix that addresses the root cause.
- **One mutation per step.** Never batch 3 fixes and restart — you won't know which one worked.
- Prefer commits over manual edits. Include a clear commit message.
- Deploy through the organ's standard path (systemd restart, service reload, etc.).
- **Verify after EVERY change** — if the fix didn't work, revert and try the next hypothesis.
- **Backoff strategy**: if a fix triggers a restart loop, apply exponential backoff before retrying:
  - 1st retry: wait 5s
  - 2nd retry: wait 30s  
  - 3rd retry: wait 120s (escalate)
- If the patch is irreversible, apply 888 HOLD before continuing.
- Log the change: `{who: <agent>, what: "patch", why: <root cause>, result: <outcome>}`

### Step 6: Postmortem + Structure Log

Close the loop with institutional memory.

- If a floor was breached, seal the postmortem to VAULT999 as witness.
- Write postmortem to `/root/INCIDENTS/<YYYY-MM-DD>-<slug>.md` with:
  - Trigger
  - Scope classification
  - Root cause and evidence
  - Fix applied (one change only)
  - Verification result
  - Prevention measures
- If the same symptom recurs within 7 days, treat it as a partial-fix pattern, not a new incident.
- **Final structured log**:
  ```
  {who: <agent>, what: "postmortem", why: <incident>, result: {
    trigger: <symptom>,
    root_cause: <finding>,
    fix: <change applied>,
    verified: <true/false>,
    prevention: <measure>,
    escrow_location: "/root/INCIDENTS/<file>.md"
  }}
  ```

## Allowed Tools

| Tool | Purpose |
|------|---------|
| `Bash` | Run system commands, health scripts, and log inspection |
| `Read` | Inspect config files, unit files, and incident logs |
| `Grep` | Search code, logs, and configs for root-cause signals |
| `mcp__arifos__arif_observe` | Federation vitals, web/entropy signals, repository state |
| `mcp__arifos__arif_memory` | Recall prior incidents, deploys, and decisions |
| `mcp__arifos__arif_measure` | Health, cost, drift, and topology checks |
| `mcp__arifos__arif_init` | Start governed constitutional session before high-risk actions |
| `mcp__arifos__arif_judge` | Request SEAL / SABAR / VOID verdict from arifOS |
| `mcp__arifos__arif_forge` | Execute bounded, SEAL-authorized changes |
| `mcp__arifos__arif_seal` | Append incident witness receipt to VAULT999 |

## Forbidden Actions

- **NEVER** patch a production organ without first sensing, scoping, and containing.
- **NEVER** apply an irreversible patch without 888 HOLD / sovereign ack and kernel SEAL.
- **NEVER** skip the postmortem for constitutional or repeated incidents.
- **NEVER** treat scope creep as normal; STOP and re-scope.
- **NEVER** suppress or omit Ω₀ (uncertainty) in incident receipts.
- Escalate to **arifOS 888_JUDGE** if a constitutional floor is breached or two floors disagree.

## Output Format

```markdown
## Skill Result: incident-triage

### Summary
One-paragraph summary of the incident, scope, root cause, and current status.

### Evidence
- Symptom: <what failed>
- Scope: <organ-only / federation-wide / constitutional / sovereign>
- Root cause: <finding with confidence>
- Containment: <what was preserved>
- Patch: <change applied or deferred>

### Recommendations
- Immediate verification step
- Prevention or monitoring improvement

### Escalations
- None / <list with owner and method>
```

## Escalation Path

| Condition | Escalate To | Method |
|-----------|-------------|--------|
| Constitutional floor tripped | arifOS 888_JUDGE | A2A verdict_request / MCP arif_judge |
| Irreversible action needed | Arif (F13 SOVEREIGN) | 888 HOLD |
| Root cause upstream (provider/OS/network) | Federation ops + A-FORGE | A2A ops channel |
| Scope creep during response | STOP + re-authorize via kernel | new ART cycle |
| Same incident within 7 days | Senior ops + postmortem review | incident registry |

---

## Fire-Time Reauthorization (WAJIB 5 — added 2026-07-19)

A decision made at *write-time* (job creation, cron entry, queue submission) is NOT automatically valid at *fire-time* (when the job runs). The world changes between scheduling and execution. Authorization must be re-judged.

### Affected surfaces

- Cron jobs
- Queued workers
- Renovate / dependency-update PRs
- Scheduled deployments
- Delayed shell jobs
- Retry queues
- Event-triggered automation (watchers firing on condition)
- Long-running MCP tasks
- PR-bot comments → actions

### The required invariant

Every deferred mutation must be judged TWICE:

```
write-time authorization
        +
fire-time authorization (re-judged at execution)
```

At fire time, the system MUST re-check:

| Check | Why |
|---|---|
| Identity and session validity | Session may have expired or been revoked |
| Lease expiry | Lease may have lapsed |
| Current branch / commit | Source of truth may have changed |
| Current target state | The world the action was queued against may have moved |
| Changed blast radius | Conditions may have widened the impact |
| New evidence | A new finding may invalidate the original decision |
| Human approval validity | Aprover may have rescinded |
| Dependency health | A package update may have failed |
| Rollback availability | The rollback path may now be broken |
| Whether the request has been revoked | Revocation must propagate to all queued instances |

### Failure behavior

A scheduled action with **expired authority MUST become HOLD, not "continue because it was approved yesterday."**

This is a **HARD kernel rule**:
- `write_time_authorization.expiry < now()` → return 888_HOLD at fire time
- Reason: `"DEFERRED_FIRE_TIME_AUTH_EXPIRED — wrote <timestamp>, expires <timestamp>, now <timestamp>"`

### Required implementation pattern

```ts
async function fireDeferredAction(deferred: DeferredEnvelope): Promise<FireResult> {
  const authCheck = await reauthorizeAtFireTime(deferred);
  if (!authCheck.valid) {
    return {
      state: "HOLD",
      reason: authCheck.reason,
      receipt: await vault.append({ type: "deferred_fire_hold", ... })
    };
  }
  return forgeExecute(authCheck.scope);
}
```

### Tests required

- Cron job queued today with `expires_at = today + 1h` → tomorrow HOLD
- Scheduled deployment after upstream service became unavailable → HOLD with explicit reason
- Revoked session → all queued actions in that session become HOLD
- Branch changed between queue and fire → re-judge blast radius, may HOLD
- Retry queue after original action's rollback path broken → HOLD

### Authority scope

WAJIB 5 is **T3 (F13 ratification required)**. This skill section documents the protocol. Implementation requires scheduler integration with the kernel judgment pipeline.

---

## Organ Disagreement Doctrine (WAJIB 7 — added 2026-07-19)

When GEOX, WEALTH, and WELL recommend incompatible actions — all internally valid — the federation needs a binding resolution order. Majority vote does not apply; three organs are not interchangeable voters.

### Resolution order (mandatory)

1. **Hard veto conditions** (any organ may trigger HOLD with evidence)
2. **Blast-radius precedence** (organ owning dominant irreversible consequence receives higher weight)
3. **Pareto search** (seek alternative satisfying all hard constraints before escalation)
4. **F13 escalation** (if no acceptable option → escalate to Arif, never silently fall back to execution order)

### Hard veto table

| Organ | May veto when | Release condition |
|---|---|---|
| GEOX | Physical infeasibility OR unacceptable earth uncertainty | New data or revised interpretation that resolves the issue |
| WELL | Unsafe human or operational readiness | Confirmed safe capacity + witness |
| WEALTH | Insolvency, unaffordable exposure, prohibited capital risk | Restructured deal OR capital limit raised |
| arifOS | Authority, law, or constitutional violation | Ratified exception OR constitutional amendment |

A veto MUST include evidence + defined release condition. Rhetorical veto is not permitted.

### Blast-radius precedence

| Dominant irreversible consequence | Owner |
|---|---|
| Subsurface irreversibility | GEOX |
| Capital survival / sovereign financial exposure | WEALTH |
| Human safety, dignity, organizational collapse | WELL |
| Constitutional / authority conflict | arifOS |

### Pareto search before escalation

Before escalating to F13, attempt:
- Smaller scope
- Delayed decision
- More evidence
- Pilot programme
- Reduced capital
- Different staffing
- Reversible experiment

### F13 escalation payload

If no acceptable Pareto option exists:

```yaml
unresolved_conflict:
  intent: <original request>
  vetoes: [{organ, reason, evidence, release_condition}, ...]
  competing_actions:
    - path: <GEOX recommendation>
      evidence: <refs>
      consequences: <list>
      reversibility: <class>
    - path: <WEALTH recommendation>
      ...
    - path: <WELL recommendation>
      ...
  recommended_least_regret: <one path with justification>
  explicit_unknowns: <list>
```

The kernel does NOT manufacture consensus. It escalates with full disagreement visible.

### Authority scope

WAJIB 7 is **T3 (F13 ratification required)**. Doctrine documentation is T1.

---

*Skill version 1.0.0 — AAA Skill Library*