muthuishere Silent-failure triage
Most production incidents are not a red alert nobody noticed. They are a green signal that was never capable of turning red. This skill is for the class of failure where every surface you own reports healthy and the thing is, nonetheless, dead.
The one rule
Before you believe any check, ask: what would this check return if the system were broken?
If the answer is "the same thing it is returning now", you have measured nothing. That check is decoration. This single question resolves more incidents than any dashboard, and it is the rule the rest of this skill is built from.
Corollaries you should say out loud while debugging:
- A 200 is proof the server answered, never proof it did the thing.
- Zero results and a broken query look identical. So do "no errors" and "not running".
- Silence is not success. A cron that stopped firing is quieter than one that works.
- A check you have never watched fail is not a check. Break it on purpose once.
How to run a triage
Work this order. Stop at the first thing that explains the symptom — and then keep going anyway for one more pass, because these failures cluster (one dead dependency commonly produces two unrelated-looking symptoms).
1. Separate liveness from delivery
/health almost always answers "is this process running". The user's complaint is almost always
"did my thing arrive". Those are different questions and the gap between them is where this whole
class lives.
Ask: what is the last END-TO-END success, with a timestamp? Not "is it up" — when did a real message reach a real recipient, a real row get written, a real webhook get accepted?
# the shape of the right probe: send a real thing, then read it back from the far side
# the shape of the wrong probe: curl /health
If you cannot answer "when did this last actually deliver", that is the bug — instrument that before you debug anything else.
2. Check the boring layers before the interesting one
Engineers debug their own code first because it is the part they can see. These are cheaper and more often the answer:
| Suspect | Fingerprint | Check |
|---|---|---|
| Quota / billing exhausted | fails in ~0s, before your config is read; fails on every request including ones that touch nothing relevant; clean flip from working to broken with no deploy | the billing page, not the logs |
| Client version rejected | server returns a generic auth/handshake error; worked for months; nothing changed on your side | compare your client version with the current release |
| Credential bound to the wrong account | it "works", to the wrong place; the label says one thing and the binding says another | print the resolved identity, not the handle/profile/alias name |
| Cert or token expired | starts at a precise instant, no deploy nearby | expiry date on the actual chain in use |
| DNS / resolver cached | recovered upstream, still failing locally; restart fixes it | resolve fresh in the failing process, not in your shell |
| Disk / inode full | writes fail, reads fine; weird unrelated errors | df -h and df -i |
| Clock skew | signature/token validation fails intermittently | compare host time to real time |
3. Ask who else holds the resource
A correct exclusive lock plus a helper process that never let go is indistinguishable from a deadlock, and reports healthy the whole time. Look for a second process on the same store, the same session, the same device registration:
# who has it open
lsof <path> # or: fuser -v <path>
ps -ef | grep <helper>
The tell: stopping the helper makes the problem vanish, and restarting it brings it back, reproducibly. If you have that, you have the answer — do not theorise further.
4. Verify the artifact, never the report of the artifact
The report of a success is not the success:
SENTin a mail API is not "the recipient's server accepted it", and it is certainly not "the From header you asked for survived". Read the sent message back and check its headers.- A deploy that returns success is not "the live URL serves this". Fetch the public URL and grep for a string that only the new version contains.
- A commit is not a deploy. A deploy is not still deployed — a later deploy silently replaces the artifact you verified.
- A test that passes is not "the code path ran". Make it fail once and watch it go red.
5. Only now, suspect your own logic
If the boring layers are clean and the artifact genuinely does not match, it is your code. By this point you will also know precisely which end-to-end assertion was missing, which is the thing worth fixing permanently.
Turn the finding into a guard
A diagnosis you do not encode will happen again — this class specialises in recurrence, because the missing alarm is the actual defect. Before you close it:
- Write the check that would have caught it.
- Break the system on purpose and watch the check go red. If you skip this you have added another green light nobody has ever seen fail, which is what caused the incident.
- Assert the resolved value, never the label: the bound email, the running version, the served bytes, the delivered message — not the handle, the intended version, the committed file, or the queued send.
What to say when you report it
State the failing fact, what a healthy system would have shown instead, and what you did to prove it. Do not report "looks fine now". Either you can name the end-to-end success with a timestamp, or the thing is still unverified — say which.
Written from production incidents, not from theory. MIT. Built by deemwar · worked, runnable examples of each class at https://examples-deemwar.pages.dev