majiayu000 reliability-engineering Skill
This skill designs systems that continue to function in the presence of faults.
1. Redundancy Patterns
- Active/Passive (Hot Standby): Primary handles load, Secondary takes over on heartbeat loss.
- Active/Active (Load Balancing): Both handle load. System survives if N-1 nodes active.
- TMR (Triple Modular Redundancy): 3 voters. 2/3 majority wins. Used in avionics/space.
2. Design for Reliability
- Watchdog Timers: Hardware/Software timer that resets the system if it hangs.
- Safety Bag / Monitor: Independent channel that monitors outputs and cuts power if limits exceeded.
- N-Version Programming: 3 teams write the same software in 3 languages (Python, Rust, Ada) to avoid common bugs.
3. Calculations
- MTBF (Mean Time Between Failures): Total Time / Number of Failures.
- Availability: MTBF / (MTBF + MTTR). Goal: 99.999% ("Five Nines" = 5 mins downtime/year).
- Failure Rate ($\lambda$): 1 / MTBF.
4. Stability Patterns
- Circuit Breaker: Stop calling a failing service to prevent cascading failure.
- Bulkhead: Isolate components so a crash in one doesn't sink the ship.
- Backpressure: Reject new work when overloaded.