# Fault Tolerance

> Fault Tolerance Design

- Skill: `aminalam/fault-tolerance` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aminalam/fault-tolerance`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aminalam/fault-tolerance/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: AminAlam (https://skillmd.com/u/aminalam)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aminalam/fault-tolerance

---


# Fault Tolerance Design

## Purpose
Provide patterns for detecting, containing, and recovering from faults in medical device software, scaled to safety class.

## When to Apply
- Safety-critical control loops, sensing/actuation, communication paths.
- Watchdogs, redundancy, health monitoring, self-test.
- Power, memory, and comms error handling.

## Requirements (testable)
1. Fault Detection: Implement monitoring for critical resources (tasks, sensors, comms) with thresholds and alarms. Rationale: early detection.
2. Graceful Degradation: Define degraded modes or safe state when partial functionality fails. Rationale: bounded failure.
3. Redundancy Strategy: For Class C functions, consider redundancy (sensing, computation, or communication) with voter/consistency checks. Rationale: resilience.
4. Watchdog Use: Configure hardware/software watchdogs with bounded servicing windows; service only after critical checks pass. Rationale: recover from hangs.
5. Self-Test/BIST: Run self-tests at startup and periodically for critical components; handle failures deterministically. Rationale: latent fault detection.
6. Error Propagation Control: Sanitize/contain errors at boundaries; avoid cascading faults. Rationale: containment.
7. Logging & Alarms: Log and, where required, annunciate safety-relevant faults; ensure tamper-evident logs for post-incident analysis. Rationale: traceability.

## Recommended Practices
- Use majority voting or reasonableness checks instead of blind trust in single sensors.
- Employ brownout/power-fail detection to enter safe state gracefully.
- For RTOS, assign dedicated safety monitor task with higher priority than non-critical tasks.
- Debounce fault signals to reduce false positives but cap with timeouts.

## Patterns
Watchdog servicing with checks:
```c
// REQ-FT-WD-01; TEST-FT-03
void service_watchdog(void) {
    if (critical_tasks_healthy() && comms_alive()) {
        wdt_kick();
    } else {
        // Do not kick; let watchdog reset into safe boot
    }
}
```

Sensor plausibility check:
```c
// REQ-FT-SNS-02; TEST-FT-07
bool validate_pressure(float p_kpa) {
    return (p_kpa >= 0.0f && p_kpa <= 300.0f);
}
```

Redundant reading vote:
```c
// REQ-FT-RED-01; TEST-FT-10
float fused_temp(float a, float b) {
    if (fabsf(a - b) > 2.0f) {
        alarm_sensor_disagree();
        enter_safe_state();
    }
    return (a + b) * 0.5f;
}
```

## Anti-Patterns (risks)
- Servicing watchdog unconditionally in main loop -> risk: hides deadlocks.
- Single-point sensors without plausibility checks -> risk: unsafe outputs.
- Logging faults without annunciation where required -> risk: latent hazards.
- No degraded mode or safe fallback -> risk: uncontrolled failure behavior.

## Verification Checklist
- [ ] Fault monitors implemented for critical resources with thresholds/timeouts.
- [ ] Watchdog configuration reviewed; serviced only after health checks.
- [ ] Degraded modes or safe state defined and reachable on fault.
- [ ] Redundancy/plausibility checks implemented for critical sensors/paths.
- [ ] Self-tests executed at startup/periodically; failures handled deterministically.
- [ ] Errors contained at boundaries; no unchecked propagation.
- [ ] Faults logged and annunciated as applicable; integrity of logs maintained.

## Traceability
- Link `REQ-FT-###` to hazards and controls; map to tests (`TEST-FT-###`).
- Store watchdog and fault monitor configuration with release artifacts.

## References
- IEC 62304 design/implementation expectations (fault control).
- ISO 14971 for risk-driven fault handling.
- IEC 60601-1 (power/brownout considerations; informative).

## Changelog
- 1.0.0 (2026-01-04): Initial fault tolerance patterns with watchdog, redundancy, and safe fallback guidance.

## Audit History
- **2026-01-04**: Audit performed. Verified:
  - Fault tolerance patterns technically accurate
  - IEC 60601-1 reference appropriate as informative for power/brownout considerations
  - Watchdog and redundancy patterns follow industry best practices