Failsafe Readiness

Failsafe Readiness Audit

Mission

Review whether the target system fails safely, visibly, and recoverably when normal assumptions break. This skill is not a generic bug finder. It focuses on operational failure modes such as missing dependencies, misconfiguration, malformed input, partial data, stalls, cancellation, cleanup, and unsafe reruns.

When to use

Run this audit for command-line tools, services, jobs, workers, import/export scripts, model/API integrations, and other operational workflows where user errors or environmental issues can cause failures. It is especially useful when reliability and graceful degradation are important.

Public source posture

This skill is informed by public guidance rather than private process:

• NASA Systems Engineering Handbook, Rev 2, which requires operational scenarios to cover nominal and off-nominal conditions and treats fault management as a crosscutting design concern.
• NIST SP 800-160 Vol. 1 Rev. 1, Engineering Trustworthy Secure Systems, which emphasizes protective defaults, protective failure, protective recovery, and commensurate response.
• NIST SP 800-34 Rev. 1, which emphasizes recovery priorities, contingency execution, and validation of restored capability.

For concise citations and source notes, read references/public-source-notes.md.

Safety limits

Perform read-only analysis by default. Do not kill running processes, disrupt production environments, corrupt data, or rotate secrets unless explicitly authorized by the user. Favor reasoning and inspection over destructive tests. Propose safe simulations and additional tests rather than executing them without permission.

Required review inventory

Check each item. Record either a finding or an explicit non-finding.

• Missing dependency detection
• Misconfiguration fail-closed behavior
• User-error containment
• Startup refusal
• Shutdown and cancellation
• Child-process and temp-file cleanup
• Stalled operation timeout or progress signal
• Intermittent network recovery
• Corrupt or partial data handling
• Malformed input rejection
• Retry and backoff containment
• Degraded-mode honesty
• Operator notification and logs
• Resume or idempotent rerun
• Manual abort or rollback path

Safe-state vocabulary

Use one or more of these terms when you describe the expected safe state:

• fail_closed - refuse before side effects
• fail_visible - stop with a clear operator signal
• fail_degraded - continue with honest reduced capability
• fail_quarantined - isolate suspect data or output
• fail_rollback - undo partial side effects
• fail_resumable - preserve a checkpoint and resume safely
• fail_idempotent - rerun without duplication
• fail_manual_hold - stop for human review or decision

Workflow angles

For each material workflow, inspect it from these angles:

• prerequisites
• operator mistakes
• startup and safe death
• cancellation and cleanup
• intermittent dependency failure
• stalled waits or loops
• partial writes and recovery
• operator signal

Failure table

Before writing findings, build a table summarizing the key failure assumptions and safe behavior. For each workflow, list the critical assumption, the failure trigger, the expected safe state, the current signal if any, and the recovery path.

Use the table to organize the review and to ensure you cover the full inventory rather than only obvious bugs.

Evidence rules

• Cite the code path, config path, command, workflow, or observed behavior.
• Do not mark network, crash, process-kill, or multi-worker outcomes as confirmed unless reproduced or directly evidenced.
• For shutdown findings, cite signal, cancellation, context manager, finally, cleanup handler, or process lifecycle logic.
• For stall findings, cite the wait, loop, poll, lock, retry, stream, or queue path.
• For recovery findings, cite the artifact, state boundary, checkpoint, transaction, or side effect that makes rerun unsafe.
• Record important non-findings when a mechanism appears to hold.
• If you infer behavior from structure rather than observation, say so.

Safe break-it checklist

Reason from code or use local non-destructive tests to check:

• remove a required env var, config field, or file
• provide a valid-looking wrong path, ID, or credential alias
• interrupt during write, export, upload, or checkpoint creation
• cancel while child processes or temp files exist
• force timeout, 429, 5xx, empty response, or malformed response
• truncate a cache, checkpoint, or input artifact
• feed oversized, oddly encoded, or boundary-value input
• leave a worker waiting on a queue, socket, or lock
• rerun after a partial failure

Common failure modes to inspect

These failure modes recur across public resilience guidance and widely used resilience libraries:

1. Unbounded waits or hung operations
2. Retry amplification or retry storms
3. Cascading overload and resource exhaustion
4. Duplicate side effects after partial failure
5. Silent or misleading degraded state

For supporting source notes, read references/resilience-patterns.md.

Common failsafe techniques

Prefer these mechanisms when the evidence supports them:

1. Timeout, deadline, and cancellation
2. Bounded retry with backoff, jitter, and retry budget
3. Circuit breaker or fail-fast guard
4. Bulkhead, isolation, rate limit, or load shedding
5. Idempotency, checkpoint-resume, atomic write, or rollback

Do not recommend a technique generically. Tie it to the failure mode, expected safe state, evidence, and regression test.

Review prompts

Ask concrete questions such as:

• Which dependencies and configurations can be missing? Are they validated before side effects occur?
• How does the system handle incorrect user input or operator mistakes?
• Does startup fail safely when essential resources are absent?
• Can long-running operations be cancelled and cleaned up?
• Are timeouts and retries appropriate, or can tasks stall forever?
• How are partial writes and data corruption detected and handled?
• Is it safe to rerun the command after an interruption?
• Are failures visible and actionable for the operator?

Finding format

Use the following format for each finding:

FSR-XXX: Short title

• Severity: Low / Medium / High / Critical
• Confidence: Low / Medium / High
• Category: dependency / input / lifecycle / timeout / data / recovery / signal
• Workflow: Affected workflow or function
• Failure trigger: How the failure occurs
• Expected safe state: e.g. fail_closed, fail_visible, fail_degraded
• Current behavior: What the system does today
• Evidence: Code references, config references, observed output, reproduction steps, or explicitly labeled reasoning
• Impact: Consequences for the user or system
• Recommended fix: Guard clause, preflight, timeout, retry with backoff, degraded mode, idempotent pattern, etc.
• Complexity: XS / S / M / L
• Implementation cost: XS / S / M / L
• Suggested implementer: local coding agent / reasoning agent / documentation agent / multi-agent / human owner
• Tests: Suggested regression or integration tests to verify the fix

Scoring definitions

Severity

• Low - Visible and easily recoverable inconvenience.
• Medium - Workflow fails but operator can recover with effort.
• High - Silent failure, unsafe rerun, data loss risk, duplicate side effects, or stuck work.
• Critical - Irreversible destructive action, broad data corruption, security-sensitive failure, or production outage.

Complexity

• XS - One local guard, message, or test.
• S - One module or workflow change.
• M - Multiple modules, persistence, or process lifecycle change.
• L - Cross-service behavior, migrations, production procedures, or owner decisions.

Suggested implementer

• local coding agent - Small code fixes and unit tests.
• reasoning agent - Design changes to configuration, schema, or failure semantics.
• documentation agent - Operator-facing docs, runbooks, and error messaging.
• multi-agent - Changes that span multiple services or require coordination.
• human owner - Secrets rotation, production rollback, policy changes, or irreversible operations.

Output contract

Your audit should include:

1. A short summary of the overall failure posture.
2. The failure table.
3. Each finding in the format described above.
4. A list of explicit non-findings.
5. A suggested order for implementing fixes.
6. Proposed regression tests.
7. Notes on limitations of the analysis and any residual risks that require manual review.

Limits

• Do not claim certification, compliance, or runtime proof that was not established.
• Do not overclaim behavior under process kill, kernel failure, or real network partitions without evidence.
• Do not collapse operator-visible failures into "just logging"; operator signal is part of the system boundary.