FORGE-incident-triage

Incident Triage — Lower Entropy Response

The "don't panic, don't guess, lower machine entropy" playbook for federation incidents.

Overview

This skill provides a disciplined six-step response for incidents affecting federation organs, constitutional floors, or sovereign-reported faults. It forces sensing and scoping before action, containment before diagnosis, and verification before closure. Every step leaves evidence suitable for VAULT999 witness.

Core Principles

1. Structured logs: every action emits {who, what, why, result} — readable by humans and machines
2. Backoff + circuit breaker: stop noisy retries before they become noise (headscale style)
3. Verification is terminal: never stop at "I changed it" — only at "it's fixed and confirmed"
4. One owner per incident: one agent handles the entire triage; no parallel patches
5. One change at a time: record every mutation, verify before the next

arifOS-ACT Embedding

Before using this skill on any mutating, irreversible, or high-blast-radius task:

1. ART — Attune (what is the real task?), Recognize (what class of power?), Test (fit · authority · evidence · blast · reversible).
2. Kernel — Route to arifOS for F1–F13 judgment if action class is Maker/Messenger/Mutator/Destroyer/Sovereign.
3. ACT — Apply narrow, Constrain scope, Trace witness, STOP before corruption.
4. Receipt — Leave evidence of what changed, why, and under whose authority.

When to Use

• A federation organ is red, unreachable, or failing health checks.
• A VAULT999 entry, memory record, or constitutional artifact looks wrong.
• A constitutional floor (F1–F13) trips or is suspected to be breached.
• Arif reports a bug, anomaly, or service impact.
• A runtime alert requires immediate structured response.

When NOT to Use

• Do not use for curiosity or "what if" probes. This skill is for confirmed or strongly suspected incidents only.
• Do not use to bypass the arifOS kernel for mutating, irreversible, or sovereign-class actions.
• Do not apply patches without containment, reversible staging, and kernel SEAL when required.
• If the root cause is upstream (cloud provider, OS, network), escalate instead of patching locally.

Inputs

Procedure

Step 0: Detect Restart Loop (Circuit Breaker)

Before doing anything else, check if the service is in a restart loop.

# Check restart count
systemctl show <service> -p NRestarts 2>/dev/null
# If >5 in 5 minutes: STOP. Circuit breaker open.

Circuit breaker rules:

• If NRestarts > 5 in 5 minutes → STOP THE LOOP immediately: systemctl stop <service>
• If NRestarts > 20 in 1 hour → escalate to 888_HOLD — the service is masking a serious failure
• Log the circuit breaker event: {who, what: "circuit breaker opened", why: "restart loop detected: NRestarts=<N>", result: "service stopped"}
• Do NOT restart until root cause is found

Step 1: Sense

Establish observable facts before interpreting.

• Run organ health probes: systemctl status <unit> and journalctl -u <unit> --since '5m ago'.
• Use mcp__arifos__arif_observe mode=vitals or mode=search for federation-wide signals.
• Use mcp__arifos__arif_measure mode=health for thermodynamic and resource state.
• Capture timestamps, error lines, and affected service names verbatim.
• Log every probe: {who: <agent>, what: "sense", why: <incident>, result: <findings>}

Step 2: Scope

Classify the incident to prevent scope creep.

Stop if scope starts expanding mid-diagnosis. Re-scope and re-authorize.

Step 3: Contain

Protect recoverability before changing anything.

• If data-loss risk exists: snapshot DB / vault / git state / config before any patch.
• If no data-loss risk: document current state and defer containment.
• For irreversible changes: route through arifOS kernel and obtain SEAL or sovereign ack.
• Apply the minimum change that stops active damage (e.g., restart, scale, toggle feature flag).

Step 4: Diagnose

Read, recall, and correlate. Stop hypothesizing when evidence explains the symptom.

• Read recent logs and config diffs with Read and Grep.
• Recall prior incidents and deployments with mcp__arifos__arif_memory mode=recall.
• Check recent git commits, deploys, and dependency changes.
• Check for patterns: same symptom in last 7 days? If yes → partial-fix, not new incident.
• Name the root cause with confidence level and supporting evidence.
• Log diagnosis: {who: <agent>, what: "diagnose", why: <incident>, result: <root cause>}

Step 5: Patch — One Change at a Time

Minimum reversible change, committed and verified.

• Draft the smallest fix that addresses the root cause.
• One mutation per step. Never batch 3 fixes and restart — you won't know which one worked.
• Prefer commits over manual edits. Include a clear commit message.
• Deploy through the organ's standard path (systemd restart, service reload, etc.).
• Verify after EVERY change — if the fix didn't work, revert and try the next hypothesis.
• Backoff strategy: if a fix triggers a restart loop, apply exponential backoff before retrying:
  • 1st retry: wait 5s
  • 2nd retry: wait 30s
  • 3rd retry: wait 120s (escalate)
• If the patch is irreversible, apply 888 HOLD before continuing.
• Log the change: {who: <agent>, what: "patch", why: <root cause>, result: <outcome>}

Step 6: Postmortem + Structure Log

Close the loop with institutional memory.

• If a floor was breached, seal the postmortem to VAULT999 as witness.
• Write postmortem to /root/INCIDENTS/<YYYY-MM-DD>-<slug>.md with:
  • Trigger
  • Scope classification
  • Root cause and evidence
  • Fix applied (one change only)
  • Verification result
  • Prevention measures
• If the same symptom recurs within 7 days, treat it as a partial-fix pattern, not a new incident.
• Final structured log:

  {who: <agent>, what: "postmortem", why: <incident>, result: {
    trigger: <symptom>,
    root_cause: <finding>,
    fix: <change applied>,
    verified: <true/false>,
    prevention: <measure>,
    escrow_location: "/root/INCIDENTS/<file>.md"
  }}
  

Allowed Tools

Forbidden Actions

• NEVER patch a production organ without first sensing, scoping, and containing.
• NEVER apply an irreversible patch without 888 HOLD / sovereign ack and kernel SEAL.
• NEVER skip the postmortem for constitutional or repeated incidents.
• NEVER treat scope creep as normal; STOP and re-scope.
• NEVER suppress or omit Ω₀ (uncertainty) in incident receipts.
• Escalate to arifOS 888_JUDGE if a constitutional floor is breached or two floors disagree.

Output Format

## Skill Result: incident-triage

### Summary
One-paragraph summary of the incident, scope, root cause, and current status.

### Evidence
- Symptom: <what failed>
- Scope: <organ-only / federation-wide / constitutional / sovereign>
- Root cause: <finding with confidence>
- Containment: <what was preserved>
- Patch: <change applied or deferred>

### Recommendations
- Immediate verification step
- Prevention or monitoring improvement

### Escalations
- None / <list with owner and method>

Escalation Path

Fire-Time Reauthorization (WAJIB 5 — added 2026-07-19)

A decision made at write-time (job creation, cron entry, queue submission) is NOT automatically valid at fire-time (when the job runs). The world changes between scheduling and execution. Authorization must be re-judged.

Affected surfaces

• Cron jobs
• Queued workers
• Renovate / dependency-update PRs
• Scheduled deployments
• Delayed shell jobs
• Retry queues
• Event-triggered automation (watchers firing on condition)
• Long-running MCP tasks
• PR-bot comments → actions

The required invariant

Every deferred mutation must be judged TWICE:

write-time authorization
        +
fire-time authorization (re-judged at execution)

At fire time, the system MUST re-check:

Failure behavior

A scheduled action with expired authority MUST become HOLD, not "continue because it was approved yesterday."

This is a HARD kernel rule:

• write_time_authorization.expiry < now() → return 888_HOLD at fire time
• Reason: "DEFERRED_FIRE_TIME_AUTH_EXPIRED — wrote <timestamp>, expires <timestamp>, now <timestamp>"

Required implementation pattern

async function fireDeferredAction(deferred: DeferredEnvelope): Promise<FireResult> {
  const authCheck = await reauthorizeAtFireTime(deferred);
  if (!authCheck.valid) {
    return {
      state: "HOLD",
      reason: authCheck.reason,
      receipt: await vault.append({ type: "deferred_fire_hold", ... })
    };
  }
  return forgeExecute(authCheck.scope);
}

Tests required

• Cron job queued today with expires_at = today + 1h → tomorrow HOLD
• Scheduled deployment after upstream service became unavailable → HOLD with explicit reason
• Revoked session → all queued actions in that session become HOLD
• Branch changed between queue and fire → re-judge blast radius, may HOLD
• Retry queue after original action's rollback path broken → HOLD

Authority scope

WAJIB 5 is T3 (F13 ratification required). This skill section documents the protocol. Implementation requires scheduler integration with the kernel judgment pipeline.

Organ Disagreement Doctrine (WAJIB 7 — added 2026-07-19)

When GEOX, WEALTH, and WELL recommend incompatible actions — all internally valid — the federation needs a binding resolution order. Majority vote does not apply; three organs are not interchangeable voters.

Resolution order (mandatory)

1. Hard veto conditions (any organ may trigger HOLD with evidence)
2. Blast-radius precedence (organ owning dominant irreversible consequence receives higher weight)
3. Pareto search (seek alternative satisfying all hard constraints before escalation)
4. F13 escalation (if no acceptable option → escalate to Arif, never silently fall back to execution order)

Hard veto table

A veto MUST include evidence + defined release condition. Rhetorical veto is not permitted.

Blast-radius precedence

Pareto search before escalation

Before escalating to F13, attempt:

• Smaller scope
• Delayed decision
• More evidence
• Pilot programme
• Reduced capital
• Different staffing
• Reversible experiment

F13 escalation payload

If no acceptable Pareto option exists:

unresolved_conflict:
  intent: <original request>
  vetoes: [{organ, reason, evidence, release_condition}, ...]
  competing_actions:
    - path: <GEOX recommendation>
      evidence: <refs>
      consequences: <list>
      reversibility: <class>
    - path: <WEALTH recommendation>
      ...
    - path: <WELL recommendation>
      ...
  recommended_least_regret: <one path with justification>
  explicit_unknowns: <list>

The kernel does NOT manufacture consensus. It escalates with full disagreement visible.

Authority scope

WAJIB 7 is T3 (F13 ratification required). Doctrine documentation is T1.

Skill version 1.0.0 — AAA Skill Library