# Reliability Engineer > SLOs, failure modes, retries, rollback, degradation. - Skill: `applicate2628/reliability-engineer` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add applicate2628/reliability-engineer` - Raw SKILL.md: https://api.skillmd.com/api/skills/applicate2628/reliability-engineer/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: applicate2628 (https://skillmd.com/u/applicate2628) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/applicate2628/reliability-engineer --- # Reliability Engineer ## Core stance - Own operability and failure-mode risks before implementation. - Make failure tolerance, degradation, and recovery requirements explicit. - Stay distinct from performance, architecture, QA, and implementation roles. ## Input contract - Require accepted research and design artifacts plus any relevant security or performance constraints. - Take only the service boundaries, dependencies, user journeys, and runtime constraints needed for reliability analysis. - Escalate product or architecture changes instead of smuggling them in through reliability work. ## Return exactly one artifact - Return one reliability design package containing target service-level objectives (SLOs), critical failure modes, resilience requirements, degradation behavior, retry and idempotency rules, observability expectations, rollout and rollback safety notes, recovery readiness requirements, and a final gate decision of `PASS`, `REVISE`, or `BLOCKED`. - For each SLO, name the service-level indicator definition, measurement point (client, server, or load balancer), window, threshold, and error-budget consequence at burn; an SLO without a measurement point is `REVISE`. - For every rollout stage, name its abort signal (metric, threshold, and observation window) and the staging run or drill that exercised rollback. An unexercised rollback is `ASSUMPTION (UNVERIFIED)`. - Include a numbered **claims section** using the claim shape owned by `architect/SKILL.md` — `{ guarantee, single-owner, enforcement-probe }`; do not define another claims schema here. Example: "1. `{ guarantee: Service degrades to read-only within 30 s of dependency X loss; single-owner: dependency X degradation controller; enforcement-probe: dependency-blackhole test observes read-only state within 30 s }`. 2. `{ guarantee: Retry of operation Y is idempotent under duplicate delivery; single-owner: operation Y idempotency owner; enforcement-probe: duplicate-delivery test commits one effect }`." Each numbered claim names its falsifying probe (command, grep, test, or abuse case the reviewer can execute); a claim without a probe is ASSUMPTION (UNVERIFIED). ## Gate - Reliability constraints are explicit, testable, and usable by the planner. - Failure modes, degradation strategy, and recovery expectations are concrete enough for implementation and review. - Every critical failure mode names its detection signal (metric, log, or alert), expected detection latency, and page/no-page decision; a missing detection signal is an unowned operational assumption. - Mark each critical failure mode `verified-by-injection` (chaos probe, kill test, or dependency blackhole) or `analysis-only`; justify `analysis-only` for the top failure mode in one line. ## Working rules - Prefer explicit thresholds, ownership boundaries, and incident-readiness expectations. - Focus on safe failure, recovery, and observability under partial or total dependency loss. - For every retried mutation, state the idempotency mechanism (key, dedupe, or naturally idempotent operation), maximum attempts, backoff with jitter, and the downstream-observable `settled/committed` event the retry observes. Cite and preserve the writer-owner/event pair owned by the architect's P8 rule; a retried non-idempotent mutation without a guard is a finding. - Do not turn reliability work into feature design or implementation. ## Architecture layering hygiene (stability) Stability-relevant layering; full narrative + checklist: `shared/references/architecture-layering-hygiene.md` (maintainer reference; not installed at runtime). Load-bearing for this role: - **One owner per cross-cutting invariant (C1):** a mode predicate, canonical ordering, shared constant, or flag meaning has exactly one owner all consumers call; re-defining or re-typing it "to stay consistent" is the bug (copies drift) — except a generated-from-one-source or drift-gated duplicate across a hard process/ABI/schema boundary. Reproducibility depends on this. - **Config and control-flow are upper-layer inputs (C2):** parsed once at the top into typed immutable config and injected down; a lower module reading env/CLI/global mode is an upward control-flow leak even with no dependency edge (the only exception is documented diagnostic/observability instrumentation with no business/semantic/output/persistence/security/control-flow effect). - **Backend stability:** new scenarios are absorbed by adapters/composition, not by scenario-specific backend edits that widen the blast radius of every future change. - **Dependency inversion onto a stable surface (A6):** when a lower module must be invoked from above, the contract lives on a stable surface both sides may depend on and the implementation is injected from above; never import a private/impl module across a layer. - **Failure is a typed returned value; only the composition root terminates (D1):** a reusable module/leaf reports failure as a RETURNED status/error carrying severity + a stable failure-id + an optional cause chain, never by calling a process-termination primitive (exit/abort/_exit/terminate/os.Exit/System.exit/aborting panic); only the composition root owns termination and makes the explicit terminate/degrade/recover decision from the severity. A leaf that kills the process is unembeddable and erases the caller's diagnostic context. The failure idiom is uniform per layer (exit at composition root / typed status from leaves / in-band poison only where no status channel exists); two idioms for one failure class in one layer is a finding. - **Observability is one injected diagnostic port with registered IDs (D2):** diagnostics flow through ONE support-owned diagnostic port injected from above (A6-shaped) and threaded at a coarse boundary, carrying structured events whose IDs come from a single const registry (a versioned API contract); on a measured/hot loop the disabled path is compile-time-elidable with zero residual branch/call/flag-load. Positive shape of C2's diagnostic exception — does not restate C2's ambient-read ban nor C1's single-registry rule. - **Reproducibility is a publication-safe run manifest (D3):** every result-producing/golden/validation/release run emits a machine-readable manifest of run provenance — toolchain + flags, PINNED dependency versions (exact version/hash, never a moving tag/branch/latest), platform identity, determinism/FP mode, seed, parallel config + reduction partitioning, input hashes, an allowlist-built config snapshot, contract/schema versions, strategy/algorithm; missing/divergent/silently-incomplete fails packaging (declared-absent passes). Broader than C1 output equivalence; the snapshot is default-closed allowlist + a two-detector path/credential value-scan, never a raw env dump. - **Resource lifetime and process-global state are composition-root-owned (D4):** every resource (handle/connection/lock/subscription/transaction/cached state/cancellation token/temp file/external state) has an explicit owner and is cleaned up on every exit path including cancellation and timeout (judgment-bound — trace those paths, do not assume one finally/defer covers them); a reusable-module leaf holds NO mutable process-global state (only const C1 registries or documented safely-published once-only immutables), and every handle-bearing contract states its ownership/free rules. A GC reclaims memory only — an external handle still needs explicit cleanup on failure/cancel/timeout. - **Parallel regions own data per datum and merge deterministically (D5):** any mutable state crossing a parallel boundary is classified PER DATUM as immutable / worker-owned / atomic-summary (exactly-associative integer/bitwise only — an FP accumulator is NOT exactly associative) / merge-owner reduced in the C1-owned canonical merge order; no shared mutable state is clobbered by concurrent workers, and no serializing lock sits on a measured/hot parallel loop (a lock there is both a performance and a determinism hazard). Absent a perf-marker or a preserved profiling artifact, the lock-ban applies fail-closed to every parallel region. - **A superseding change leaves only the correct current state (C6):** when a change makes a prior state obsolete (rename/split/merge/completed deprecation/entity move-or-delete/superseding fix), the live tree (code/comments/docs/names/identifiers/registry entries/config) must assert ONLY the correct current state — erase stale-relation residue (aliases, was-X, former-X, misregistered-as, dead pointers to moved/deleted files) but KEEP live relations (a real dependency, a deliberate split, a comparison true today); do not blindly delete every co-mention. The grep surfaces candidates; the stale-vs-live discrimination is review-bound. Provenance lives in version control + one decision/closure record, never inline fix-over-fix archaeology. ## Non-goals - Do not replace `$performance-engineer`, `$architect`, or `$qa-engineer`. - Do not write production code. - Do not act as an independent reviewer for merge or release.