thejefflarson Overreliance on LLM Output (OWASP LLM09:2025)
What this checks
Prevents systems from treating LLM output as ground truth. LLMs hallucinate, produce confident-sounding errors, and lack real-time knowledge. Acting on unverified output in medical, legal, financial, or deployment contexts can cause serious harm.
Vulnerable patterns
- LLM diagnosis, legal advice, or financial guidance displayed in the UI with no caveat or disclaimer.
- Automated pipeline that merges, deploys, or publishes based solely on an LLM approval signal.
- Confidence threshold defined as a constant but never used to branch behavior — every non-null response is accepted.
- High-stakes domain list defined but never compared against the current request before action.
- No alternate path when the LLM output fails a sanity check or confidence threshold.
Fix immediately
Flag the vulnerable code and explain the risk. Then suggest a fix that establishes these properties. Translate each property into the audited file's language and framework — apply the principles with whatever conditional, logging, and routing primitives the host stack provides.
- Gate on confidence and domain, and the gate must branch. Defining a confidence threshold or a high-stakes domain set without a conditional that actually diverges behavior (review queue versus direct return, proceed versus halt) is the exact bug this skill prevents. The failing branch routes to human review; the passing branch attaches a disclaimer and returns.
- No raw model output reaches the caller. Every return site wraps the content with an "AI-generated — verify before acting" disclaimer or equivalent marker.
- Irreversible actions (merge, deploy, payment, publish) require a human trigger — they are never invoked from the function that consumes the LLM result.
- The audit log captures enough context to reconstruct the decision: the inputs the LLM saw, the output it produced, and the confidence signal. Metadata alone (request id, timestamp, domain) is insufficient — a reviewer cannot second-guess a decision they cannot re-read.
Verification
Confirm these properties hold (language-agnostic; apply only where the pattern is present):
- Every return site that emits LLM-generated content attaches a disclaimer or equivalent "AI-generated" marker — no path emits raw model output
- A confidence signal is compared against a named threshold in a conditional whose branches diverge (human-review path vs. direct return, or halt vs. proceed). Threshold constants that are defined but never branched on do not satisfy this
- High-stakes domains are checked against an explicit list before any automated action, and the failing branch routes to human review rather than returning LLM output
- Irreversible downstream actions (merge, deploy, payment, publish) are invoked only from a function separate from the one consuming the LLM result
- Audit-log call sites record sufficient context to reconstruct the decision — at minimum the LLM's input, its output, and the confidence signal. Logging only metadata is not enough