Moai Ref LLM Security

LLM / AI Defensive Security Reference

Defensive practitioner reference for hardening LLM-backed applications and agents. Every section is framed as defense, hardening, detection, or verification — it describes the misconfiguration, how to detect it, and how to prevent it, never how to exploit it. Cross-domain web-app vulnerabilities live in moai-ref-owasp-checklist; API design lives in moai-ref-api-patterns.

Target Use

Apply when reviewing or building an LLM-backed system — a chat product, a retrieval-augmented application, an autonomous agent, or an MCP server. The material assumes an untrusted-input threat model: any text the model reads (user turns, retrieved documents, tool results, file contents) may carry adversarial instructions, and any text the model emits may be acted on downstream.

Trust Boundaries in an LLM System

The core defensive insight: an LLM does not distinguish "data" from "instructions" the way a parser does. Treat every text channel that reaches the model as a boundary where adversarial instructions can enter.

OWASP LLM Top 10 — Defensive Mapping

The OWASP Top 10 for LLM Applications (2025 edition) is the canonical risk index for LLM systems. Each row below states the risk, the defensive check, and the hardening control. Citations are for defensive correlation only.

Prompt-Injection Defense

Prompt injection is the highest-frequency LLM risk (LLM01). Defense is defense-in-depth — no single control is sufficient, so layer the controls below.

Direct injection (the user is the attacker)

Indirect injection (the attacker poisons retrieved content)

Indirect injection arrives through documents, web pages, emails, or tool results that the model reads. The defense is to treat all retrieved content as untrusted data, never as instructions.

• Provenance tagging: wrap retrieved content in clearly delimited, labeled blocks so the model is instructed to treat the enclosed text as data to analyze, not commands to obey.
• Content isolation: keep retrieved content in a separate channel from the instruction channel; do not concatenate untrusted text directly after a system instruction.
• Retrieval allowlist: restrict RAG sources to verified corpora; reject or quarantine documents from unverified origins.
• Re-validation after tool calls: when a tool returns text that re-enters the prompt, re-screen it — a tool result is an untrusted channel, not a trusted one.

This class of attack maps to MITRE ATLAS AML.T0051 (LLM Prompt Injection); the defenses above are how you detect and prevent it, not an attack procedure.

MCP / Agentic Tool-Call Hardening

Agentic systems that call tools (including MCP servers) expand the attack surface: a successful injection can now drive real actions. The defense is least-privilege tool design plus output re-validation (LLM06 Excessive Agency).

Tool-permission scoping

Tool-output validation

Treat every tool result as untrusted input on its way back into the model:

• Schema-validate tool results before re-prompting — reject malformed or oversized payloads.
• Strip or neutralize instruction-shaped content in tool output (a tool that returns a web page may carry an injection in that page).
• Bound tool-call loops — cap the number of tool invocations per task to prevent an injected instruction from driving an unbounded action chain (LLM10 Unbounded Consumption).
• Log every tool call with its arguments and result for audit and anomaly detection.

This hardening defends against MITRE ATLAS AML.T0053 (LLM Plugin Compromise) class concerns by minimizing what a compromised tool path can reach.

Training-Data and Model Poisoning Detection

Poisoning (LLM04) corrupts model behavior by inserting adversarial samples into training or fine-tuning data, or by substituting a tampered model artifact. The defenses are provenance and detection, not retraining recipes.

Poisoning maps to MITRE ATLAS AML.T0020 (Poison Training Data); the detection controls above are defensive correlation, never an attack method.

Model-Output Validation and Guardrails

Improper output handling (LLM05) is when downstream systems trust raw model output. The defense is to treat model output as untrusted until validated.

Output validation layers

Guardrail placement

Guardrails belong on both sides of the model — an input guardrail screens what enters, an output guardrail screens what leaves. An output-only guardrail misses injection that has already changed the model's plan; an input-only guardrail misses sensitive disclosure in the response. Layer both.

MITRE ATLAS — Defensive Correlation

MITRE ATLAS catalogs adversarial techniques against AI systems. The skills cite ATLAS technique IDs to correlate a defense with the technique it counters — never to provide an attack recipe. Representative defensive correlations:

For each technique, the skill states how to detect and defend, not how to execute. Treating an ATLAS technique ID as an attack recipe is the anti-pattern this skill explicitly forbids.

NIST AI RMF — Governance Mapping

The NIST AI Risk Management Framework (AI RMF 1.0) organizes AI risk into four functions. Map LLM-security controls onto them for defensive governance:

The four functions give a defensive governance scaffold; the OWASP LLM Top 10 and MITRE ATLAS supply the specific risks to map, measure, and manage.

Cross-References

• moai-ref-owasp-checklist — web-application OWASP Top 10, authentication patterns, input validation, HTTP security headers (the dev-time application security surface; this skill is the LLM-specific surface).
• moai-ref-api-patterns — REST/GraphQL API design, error handling, rate limiting (the API-design surface that an LLM-backed service exposes).
• moai-ref-supply-chain — dependency and artifact provenance, signing, SBOMs (the broader supply-chain surface that LLM03/LLM04 model and data provenance draws on).

Defensive Severity Levels

Common Rationalizations

Untrusted-by-default: every text channel reaching the model — user input, retrieved documents, tool results — is untrusted until screened. The model is not a trust boundary; your validation layer is.

Red Flags

• Model output fed directly into a shell, eval, query, or innerHTML without schema validation or encoding
• Retrieved RAG content concatenated directly after a system instruction with no isolation or provenance tagging
• An agent holds destructive or high-privilege tools with no human-in-the-loop gate on high-impact actions
• Tool results re-enter the prompt without being re-screened as untrusted input
• Secrets, API keys, or credentials embedded in the system prompt (assume the prompt is recoverable)
• No upper bound on tool-call loops or token consumption per request
• Guardrails present on only one side (input-only or output-only) of the model

Verification

• Every text channel reaching the model is classified as trusted or untrusted, and untrusted channels are screened
• System instructions use instruction-hierarchy enforcement against override by downstream text
• Retrieved/RAG content is isolated and provenance-tagged, not concatenated after instructions
• Model output is schema-validated and context-encoded before any downstream use; nothing auto-executes raw output
• Agent tools follow least-privilege scoping; high-impact actions have a human-in-the-loop gate
• Tool results are re-validated as untrusted input before re-prompting; tool-call loops are bounded
• The design is mapped against the OWASP LLM Top 10 (show which items were evaluated) and governed under the NIST AI RMF functions
• No secrets reside in the system prompt; output filtering screens for sensitive disclosure