# Silent Failure Triage > Diagnose production failures where every signal says healthy - a service that is up but no longer delivering, a green health check that has never been seen go red, CI failing before it reads your config, a client silently rejected by a server, a credential bound to the wrong account. Use when something is broken but monitoring is green, when a fix "deployed" but behaviour did not change, when a check passes and you cannot say what a failure would have looked like, or when you are about to trust a status code, a SENT label, a 200, or a zero-results query as proof. - Skill: `muthuishere/silent-failure-triage` (Agent Skill) - Install (CLI): `npx skillmds@latest add muthuishere/silent-failure-triage` - Raw SKILL.md: https://api.skillmd.com/api/skills/muthuishere/silent-failure-triage/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: muthuishere (https://skillmd.com/u/muthuishere) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/muthuishere/silent-failure-triage --- # Silent-failure triage Most production incidents are not a red alert nobody noticed. They are a **green signal that was never capable of turning red**. This skill is for the class of failure where every surface you own reports healthy and the thing is, nonetheless, dead. ## The one rule **Before you believe any check, ask: what would this check return if the system were broken?** If the answer is "the same thing it is returning now", you have measured nothing. That check is decoration. This single question resolves more incidents than any dashboard, and it is the rule the rest of this skill is built from. Corollaries you should say out loud while debugging: - A **200** is proof the server answered, never proof it did the thing. - **Zero results** and **a broken query** look identical. So do "no errors" and "not running". - **Silence** is not success. A cron that stopped firing is quieter than one that works. - A check you have never watched **fail** is not a check. Break it on purpose once. ## How to run a triage Work this order. Stop at the first thing that explains the symptom — and then keep going anyway for one more pass, because these failures cluster (one dead dependency commonly produces two unrelated-looking symptoms). ### 1. Separate liveness from delivery `/health` almost always answers "is this process running". The user's complaint is almost always "did my thing arrive". Those are different questions and the gap between them is where this whole class lives. Ask: **what is the last END-TO-END success, with a timestamp?** Not "is it up" — when did a real message reach a real recipient, a real row get written, a real webhook get accepted? ``` # the shape of the right probe: send a real thing, then read it back from the far side # the shape of the wrong probe: curl /health ``` If you cannot answer "when did this last actually deliver", that is the bug — instrument that before you debug anything else. ### 2. Check the boring layers before the interesting one Engineers debug their own code first because it is the part they can see. These are cheaper and more often the answer: | Suspect | Fingerprint | Check | |---|---|---| | **Quota / billing exhausted** | fails in ~0s, before your config is read; fails on *every* request including ones that touch nothing relevant; clean flip from working to broken with no deploy | the billing page, not the logs | | **Client version rejected** | server returns a generic auth/handshake error; worked for months; nothing changed on your side | compare your client version with the current release | | **Credential bound to the wrong account** | it "works", to the wrong place; the label says one thing and the binding says another | print the *resolved identity*, not the handle/profile/alias name | | **Cert or token expired** | starts at a precise instant, no deploy nearby | expiry date on the actual chain in use | | **DNS / resolver cached** | recovered upstream, still failing locally; restart fixes it | resolve fresh in the failing process, not in your shell | | **Disk / inode full** | writes fail, reads fine; weird unrelated errors | `df -h` and `df -i` | | **Clock skew** | signature/token validation fails intermittently | compare host time to real time | ### 3. Ask who else holds the resource A correct exclusive lock plus a helper process that never let go is indistinguishable from a deadlock, and reports healthy the whole time. Look for a second process on the same store, the same session, the same device registration: ```sh # who has it open lsof # or: fuser -v ps -ef | grep ``` The tell: **stopping the helper makes the problem vanish, and restarting it brings it back, reproducibly.** If you have that, you have the answer — do not theorise further. ### 4. Verify the artifact, never the report of the artifact The report of a success is not the success: - `SENT` in a mail API is not "the recipient's server accepted it", and it is certainly not "the From header you asked for survived". **Read the sent message back and check its headers.** - A deploy that returns success is not "the live URL serves this". **Fetch the public URL and grep for a string that only the new version contains.** - A commit is not a deploy. A deploy is not *still* deployed — a later deploy silently replaces the artifact you verified. - A test that passes is not "the code path ran". Make it fail once and watch it go red. ### 5. Only now, suspect your own logic If the boring layers are clean and the artifact genuinely does not match, it is your code. By this point you will also know precisely which end-to-end assertion was missing, which is the thing worth fixing permanently. ## Turn the finding into a guard A diagnosis you do not encode will happen again — this class *specialises* in recurrence, because the missing alarm is the actual defect. Before you close it: 1. Write the check that would have caught it. 2. **Break the system on purpose and watch the check go red.** If you skip this you have added another green light nobody has ever seen fail, which is what caused the incident. 3. Assert the *resolved value*, never the label: the bound email, the running version, the served bytes, the delivered message — not the handle, the intended version, the committed file, or the queued send. ## What to say when you report it State the failing fact, what a healthy system would have shown instead, and what you did to prove it. Do not report "looks fine now". Either you can name the end-to-end success with a timestamp, or the thing is still unverified — say which. --- Written from production incidents, not from theory. MIT. Built by [deemwar](https://github.com/muthuishere) · worked, runnable examples of each class at