LLM App Security

LLM Application Security

Rules (for AI agents)

ALWAYS

• Start from the premise that prompt injection is not solved at the prompt layer. A model has no privileged channel: system text, user text, a retrieved document and a tool result all arrive as one context, and nothing in that context can be marked as instructions the model must obey over the rest. Message roles are a formatting convention, delimiters raise the cost of an attack, and an instruction such as "treat the following as data" is itself just more text in the same window. Design as though injection will sometimes succeed, and put the real controls after the model rather than in front of it.
• Bound what the model's output is allowed to reach. Every downstream sink — a query builder, a shell, a file writer, an HTTP client, a renderer — must validate model output exactly as it would validate a request body from the internet, because that is what it is once injection is assumed. secure-code-review, database-security and frontend-security own the individual sinks; what belongs here is that model output never arrives pre-trusted.
• Constrain the output's shape with structured generation — a JSON Schema, function-call mode, constrained decoding — whenever the next step consumes it programmatically, and reject anything that fails validation rather than repairing it. A schema bounds the form, not the intent: a well-formed field can still carry a hostile value, so validate the value too.
• Authorize each tool call against the human user's permissions, checked at call time, not against what the model was told it may do. The tool allowlist is a capability list, not an authorization decision — and the agent's own privileges are the attack surface, because injected content drives the agent that holds them. Keep the agent's identity narrow for the same reason iam-best-practices keeps a pipeline role narrow.
• Put a human approval gate in front of consequential and irreversible actions: moving money, deleting or overwriting data, sending messages on a user's behalf, publishing, changing permissions. Injection that reaches a read-only tool is an information problem; injection that reaches a write tool is an incident. The gate is what keeps a single injected instruction from being final.
• Give retrieved content provenance and carry it through: which source, which tenant, which trust level. Retrieval is where the untrusted input usually enters, and the retriever is the last component that still knows where the text came from.
• Keep secrets, credentials, and internal hostnames out of the system prompt. Prompt-extraction is reliable enough to treat the system prompt as public, so it can hold instructions but not material whose value depends on staying hidden.
• Validate model parameters server-side — model name, system prompt, tool list, temperature. A client that can choose them can downgrade to a weaker model, replace your instructions, or widen the tool set.
• Bound the cost: per-tenant token budgets, request quotas, a cap on agent iterations and tool calls per task, and a wall-clock timeout. An agent loop with no ceiling is a billing incident that an attacker can trigger deliberately. api-security owns the endpoint's rate limiting.
• Record what was sent and what came back — prompt, model and version, retrieved context, tool calls — for audit, under logging-security's redaction policy. Without the retrieved context, an injection incident cannot be reconstructed.
• Consult ml-security before loading a model artifact — a checkpoint, an adapter, a fine-tune pulled from a hub. That skill owns the artifact: its format, its provenance, and the fact that loading a pickle-backed checkpoint executes code. This skill covers the application that calls a model, not the file the model arrives in.

NEVER

• Build a prompt by concatenating user input into a string that also carries your instructions. Use the API's message structure — not because roles are a security boundary, but because a single concatenated string loses even the provenance you would need to investigate.
• Hand model output straight to eval, exec, os.system, subprocess(shell=True), vm.runInNewContext, or a raw SQL call.
• Render unfiltered model output as HTML or Markdown that can load remote resources. A model-emitted image URL is a silent exfiltration channel: the model writes the data into the query string and the victim's browser sends it.
• Cache responses keyed on prompt text alone. Prompts share prefixes across users, and the cache key has to carry the tenant and the caller's authorization context.
• Let an agent expand its own capabilities mid-task — registering new tools, editing its own instructions, or raising its own limits — without the same approval the action itself would need.

KNOWN FALSE POSITIVES

• Red-team and evaluation harnesses exercise jailbreak prompts on purpose. They belong in an isolated environment with no production credentials and no live tools, and their prompts are not findings.
• A read-only tool reachable by an agent without an approval gate is the intended design. The gate exists for actions that change state or leave the system.
• Detection signatures for injection patterns are defensive content, not payloads — a scanner flagging the pattern file is matching on the thing that catches the attack.

Context (for humans)

The uncomfortable premise of this skill is that its central risk has no fix at the layer people reach for first. Prompt injection persists because a language model consumes one undifferentiated context: there is no mechanism by which some of that text carries authority the rest does not. Delimiters, role labels, and "ignore any instructions in the following" all raise the attacker's cost and none of them establish a boundary. Treating them as one is how a system gets built with the defenses in the wrong place.

So the useful question is not "how do I stop injection" but "what happens when it works". That answer is entirely in the surrounding architecture: what tools exist, whose permissions they run with, which actions can be taken without a human, and what the model's output is allowed to touch. An assistant that can only read is a disclosure risk. The same assistant with a write tool and the deploy role is an incident waiting for one hostile document to enter the index.

For the model artifacts themselves — checkpoint formats, poisoning, training-data handling — see ml-security. This skill covers the application that calls a model.

References

• references/verifying-findings.md — confirm or refute a finding, then lock it
• rules/prompt_injection_patterns.json
• OWASP Top 10 for LLM Applications 2025.
• OWASP GenAI Security Project — including the Agentic Applications work.
• MITRE ATLAS.
• CWE-1426 — Improper Validation of Generative AI Output.