Production Failure Modes And Tradeoffs

Production Failure Modes & the Inference-Stack Tradeoff Map

This is the capstone. The four sibling skills each own one slice of the stack — harness-and-context-engineering (the agent loop, prompts, tools, context packing), rag-architecture (retrieval, grounding, indexing), inference-caching-and-kv (serving, prefix/KV cache, throughput), quantization-and-model-compression (weight/activation precision, model size). This skill ties them together at the layer where they all meet: what breaks in production, and what every fix costs you elsewhere.

The thesis: LLM systems fail silently

A traditional service fails loud — a 500, a stack trace, a thrown exception, an alert. An LLM system mostly fails quiet. The HTTP 200 still comes back. The JSON still parses. The agent still finishes. The answer is just worse — subtly wrong, stale, refused, hallucinated, or off-policy — and nothing in your exception tracker fires.

Three consequences drive everything below:

1. Detection is a first-class concern, co-equal with mitigation. A failure you can't see is a failure you can't fix. Every entry in the taxonomy below names how you detect it, because the default — wait for a user to complain — is unacceptable in production.
2. Evals + monitoring are the safety net that exceptions can't be. Unit tests check code paths that didn't change; only an eval suite + production sampling catch the regression where the code is identical and the output degraded.
3. Every mitigation is a trade. There is no free lunch in this stack. A retry buys reliability and spends latency + cost. A guardrail buys safety and spends latency. The second half of this skill maps those trades so you choose them on purpose.

Part 1 — The Failure Taxonomy

Eighteen modes in six categories. Columns: symptom (what you observe) · root cause · detection (the signal that catches it) · mitigation · owner skill (where the deep fix lives). "Owner" is a routing hint, not a hard boundary — most real incidents span two.

1. Structured output & tool calling

The model is a text generator wearing a structured-output costume. It will return something shaped like JSON or a tool call even when it has no business doing so. The discipline: never trust the shape — validate it at the boundary, repair once, and reject hallucinated calls before they execute. A tool call that runs delete_account with fabricated args is a production incident, not a parse error.

Do validate name + args against the live tool registry before dispatch; cap the repair loop at one retry then fail loud; track parse-failure and schema-violation rates as metrics. Don't eval() or blindly execute model output; don't retry-repair forever (a budget blowup in disguise); don't let a side-effecting tool run on unvalidated args.

2. Retrieval & context

These three rhyme: the model's answer is only as good as what's in the window, and the window is filled by retrieval + history packing. Garbage in → confident garbage out, with no error. The leverage is upstream: fix recall and ordering before you touch the generation prompt. Raising temperature or swapping models won't save you from a context that lacks the answer.

Do measure retrieval quality with a labeled set (hit@k, nDCG) and groundedness separately from end answer quality — they fail independently. Don't "fix hallucination" by lowering temperature when the real defect is the answer was never retrieved; don't equate a bigger context window with better grounding — past the effective window, more tokens hurt.

3. Agent control

An agent is a loop with a wallet. The only thing standing between a stuck plan and a $4,000 overnight bill is a hard ceiling you set in advance. Budgets are not optional config — they're the load-bearing safety mechanism of any autonomous loop.

Do set max-steps and max-cost per run, surface tool errors back into the loop so the model can adapt, and return the best partial result on abort. Don't swallow tool errors (the model loops blind), don't budget per-call instead of per-run, and don't let an agent spawn sub-agents without inheriting the parent's remaining budget.

4. Serving & infrastructure

This is the category traditional SRE intuition handles best — and the one teams under-instrument because "the model works on my laptop." In production the failures are about load and the network: a third party rate-limits you, a long prompt blows up prefill, an unstable prefix silently halves your cache hit-rate. Measure tails (p95/p99), not means, and attribute cost per request — averages hide the incident.

Do retry only idempotent calls with jittered backoff, trip a circuit breaker on sustained upstream errors, and keep a second provider/model wired as a fallback. Don't retry a side-effecting call blindly (you double-charge the card), don't alert on the mean latency (the p99 incident hides under it), and don't put a timestamp at the top of your prompt — it poisons the whole prefix cache for one token's worth of "freshness."

5. Multi-tenancy & security

LLMs erase the boundary between instructions and data — that's the whole vulnerability class. Any text in the window, from any source, can read like a command. Combine that with shared caches/indexes/memory and you get the two scariest production incidents: one tenant reading another's data, and a retrieved web page hijacking your agent. Treat all retrieved, tool-returned, and user-supplied text as untrusted data, and namespace every shared store by tenant.

Do put the tenant ID in every cache, vector-index, memory, and KV key; quote untrusted spans and label them "data, not instructions"; gate high-risk tools behind human approval. Don't key a cache on prompt text alone (collides across tenants), don't let retrieved documents carry instructions the model obeys, and don't log raw prompts that may contain secrets or PII — redact at the boundary, both directions.

6. Quality & behavior drift

This is the category that defines the silent-failure problem. Nothing throws. The code is byte-identical. The output just got worse — a provider swapped the model under your alias, a prompt edit regressed an edge case, the safety tuning crept toward refusing valid requests. The only instrument that sees these is an eval suite run against a baseline. If you ship LLM changes without one, you are flying blind by definition.

Do pin exact model-version strings, gate every prompt/model change on a no-regression eval, and track refusal-rate as a metric on a benign set. Don't point at a floating alias like latest in production (the provider moves it under you), don't assert eval expectations on exact output strings (semantic/tolerance assertions survive nondeterminism), and don't ship a prompt edit on the strength of three good spot-checks — that's how silent regressions get in.

Part 2 — The Tradeoff Map

Every production decision moves along four axes. You cannot maximize all four — pushing one usually spends another. Hold the axes in your head and name the tension before you pull a lever.

• Latency — TTFT + tokens/sec, p95/p99, not just the mean.
• Quality — task success / groundedness / eval score, not "looks good."
• Cost — $ per resolved task, attributed per tenant/feature.
• Reliability — success rate under failure: outages, ratelimits, bad inputs, load.

The one rule: before pulling any lever, say out loud which axis you're spending. "I'm adding a reranker — spending latency + cost to buy retrieval quality." If you can't name the cost, you haven't understood the lever.

Part 3 — Production Readiness Checklist

Ship nothing until each line is a verifiable "yes." Each maps to a failure mode above.

• Budgets set — per-request + per-run token/cost ceilings; agents have a max-steps cap that aborts and returns partial. (cost blowup, runaway agent)
• Evals + regression gate — golden set + LLM-judge in CI; every prompt/model change blocked on a no-regression check. (silent eval regression, version drift)
• Fallback + circuit breaker — second provider/model wired; breaker opens on sustained errors; retries are jittered and idempotency-gated. (provider outage, ratelimit)
• Observability + cost attribution — TTFT/decode/p95/p99, error rate, token + $ per request attributed by tenant/feature; cost-per-resolved-task tracked. (latency spike, cost blowup)
• Injection defense + tenant isolation — trusted/untrusted prompt regions separated; tool/retrieved content treated as data; every cache/index/memory key namespaced by tenant. (prompt injection, cross-tenant contamination)
• Structured-output repair — schema-validate every output/tool call at the boundary; one-shot repair loop; reject hallucinated tools before dispatch. (malformed JSON, schema violation, hallucinated tool call)
• Cache prefix stable — static system prompt + tools first, volatile content last; cache hit-rate monitored. (cache miss)
• Degraded-mode UX — define what the user sees when the model/retrieval/provider is down or slow: cached answer, smaller model, "try again," graceful partial — never a hang or a silent wrong answer. (outage, latency spike)
• Incident runbook — written, tested order of checks (below); on-call knows where the dashboards and rollback button are. (all)

Part 4 — Incident Response for LLM Systems

When an LLM system "feels worse," there's no stack trace pointing at the line. Triage by ruling out the loud, fast, reversible causes before the subtle ones.

What to check first — in order:

1. Did anything ship? Prompt, model version, RAG index, tool schema, retrieval params, config. A change is the likeliest cause. Check deploy/commit log against the symptom's start time. → fastest fix is often rollback (see canary below).
2. Is the provider healthy? Error rate, 429s, latency, status page. Outage/ratelimit is loud and external — rule it out in seconds before chasing quality.
3. Did the inputs change? New tenant, new traffic pattern, a payload that blew the context window, a doc that injected the prompt. Sample the actual failing requests.
4. Did retrieval degrade? Stale index, recall drop, empty results. Log + replay the retrieved chunks for a failing query.
5. Did quality drift with no change? Run the eval suite now vs the baseline. If code is identical and eval scores dropped, suspect silent model-version drift or input drift.
6. Is it cost or latency, not quality? Split TTFT vs decode, check cache hit-rate and concurrency — that points at serving, not the model.

Worked example — "support answers went bad overnight, no alerts fired." (1) Deploy log: nothing shipped on your side. (2) Provider status: green, error rate flat. (3) Inputs: traffic normal. (4) Retrieval: replay a failing query — chunks look fine. (5) Eval vs baseline: groundedness dropped 12 points on the identical prompt + pinned code. Root cause: model-version drift — the provider rotated the model behind your alias. Fix: pin the exact prior version, open a ticket to re-eval the new one, canary it later. Note the order — you ruled out the four loud/cheap causes in minutes and only then ran the eval that named the quiet one.

Canary & rollback for prompt + model changes. Treat a prompt or model change like a code deploy, because it is one:

• Gate on the eval suite before it leaves CI.
• Canary to a small traffic % and compare live quality/cost/latency to the control before full rollout.
• Pin exact versions so "rollback" is a config flip, not an archaeology project.
• Keep the previous prompt + model pinned and one switch away — the rollback path is the whole point of canarying.

The silent-failure problem, restated. Exceptions catch the loud failures — the 5xx, the parse error, the timeout. They are blind to the quiet ones — the answer that got worse, the retrieval that went stale, the model that drifted on upgrade, the over-refusal that crept in. Those only show up in what monitoring + evals measure: groundedness, refusal rate, eval score vs baseline, cost-per-task, cache hit-rate, retrieval quality. Build those signals first. A system that can only see its loud failures is a system that's mostly blind.

Cross-references — where each fix lives

• harness-and-context-engineering — the agent loop, tool registry + validation, context packing/compression, prompt structure, injection boundaries. Owns: hallucinated tool calls, malformed/truncated JSON, schema violations, context overflow/lost-in-the-middle, runaway agents, prompt injection.
• rag-architecture — chunking, embeddings, reranking, index freshness, groundedness, citation, index namespacing. Owns: stale/irrelevant retrieval, hallucination-from-bad-context, indirect injection via documents.
• inference-caching-and-kv — prefix/KV cache, cache-key design, batching/throughput, serving latency, tenant cache isolation. Owns: cache misses (unstable prefix), latency spikes, cross-tenant cache contamination; supports cost control.
• quantization-and-model-compression — precision, model size, the quality/throughput/cost trade and its nondeterminism. Owns: the quantization + smaller-model levers; contributes to latency, cost, and non-determinism.
• this skill — synthesis: the cross-cutting failures (cost blowups, outages, over-refusal, silent eval regression, version drift, PII leakage, non-determinism), the four-axis tradeoff map, the readiness checklist, and incident response.

If you remember one thing: LLM systems fail quietly, so build the detection before the mitigation, and name the axis you spend before you pull any lever.