# LLM App Security > Securing a feature that calls an LLM: prompt injection as an unsolved input problem, bounding what model output is allowed to reach, tool authorization against the human rather than the model, approval gates on consequential actions, RAG context provenance, system-prompt leakage, and cost limits. Use when sending prompts to an LLM API, building a RAG pipeline, giving a model tools, rendering or executing model output, or exposing an LLM-backed endpoint. - Skill: `shieldnet-360/llm-app-security` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add shieldnet-360/llm-app-security` - Raw SKILL.md: https://api.skillmd.com/api/skills/shieldnet-360/llm-app-security/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: ShieldNet-360 (https://skillmd.com/u/shieldnet-360) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/shieldnet-360/llm-app-security --- # LLM Application Security ## Rules (for AI agents) ### ALWAYS - Start from the premise that **prompt injection is not solved at the prompt layer**. A model has no privileged channel: system text, user text, a retrieved document and a tool result all arrive as one context, and nothing in that context can be marked as instructions the model must obey over the rest. Message roles are a formatting convention, delimiters raise the cost of an attack, and an instruction such as "treat the following as data" is itself just more text in the same window. Design as though injection will sometimes succeed, and put the real controls after the model rather than in front of it. - Bound what the model's output is **allowed to reach**. Every downstream sink — a query builder, a shell, a file writer, an HTTP client, a renderer — must validate model output exactly as it would validate a request body from the internet, because that is what it is once injection is assumed. `secure-code-review`, `database-security` and `frontend-security` own the individual sinks; what belongs here is that model output never arrives pre-trusted. - Constrain the output's **shape** with structured generation — a JSON Schema, function-call mode, constrained decoding — whenever the next step consumes it programmatically, and reject anything that fails validation rather than repairing it. A schema bounds the form, not the intent: a well-formed field can still carry a hostile value, so validate the value too. - Authorize each tool call against the **human user's** permissions, checked at call time, not against what the model was told it may do. The tool allowlist is a capability list, not an authorization decision — and the agent's own privileges are the attack surface, because injected content drives the agent that holds them. Keep the agent's identity narrow for the same reason `iam-best-practices` keeps a pipeline role narrow. - Put a **human approval gate** in front of consequential and irreversible actions: moving money, deleting or overwriting data, sending messages on a user's behalf, publishing, changing permissions. Injection that reaches a read-only tool is an information problem; injection that reaches a write tool is an incident. The gate is what keeps a single injected instruction from being final. - Give retrieved content **provenance** and carry it through: which source, which tenant, which trust level. Retrieval is where the untrusted input usually enters, and the retriever is the last component that still knows where the text came from. - Keep secrets, credentials, and internal hostnames out of the system prompt. Prompt-extraction is reliable enough to treat the system prompt as public, so it can hold instructions but not material whose value depends on staying hidden. - Validate model parameters server-side — model name, system prompt, tool list, temperature. A client that can choose them can downgrade to a weaker model, replace your instructions, or widen the tool set. - Bound the cost: per-tenant token budgets, request quotas, a cap on agent iterations and tool calls per task, and a wall-clock timeout. An agent loop with no ceiling is a billing incident that an attacker can trigger deliberately. `api-security` owns the endpoint's rate limiting. - Record what was sent and what came back — prompt, model and version, retrieved context, tool calls — for audit, under `logging-security`'s redaction policy. Without the retrieved context, an injection incident cannot be reconstructed. - Consult `ml-security` before loading a model artifact — a checkpoint, an adapter, a fine-tune pulled from a hub. That skill owns the artifact: its format, its provenance, and the fact that loading a pickle-backed checkpoint executes code. This skill covers the application that calls a model, not the file the model arrives in. ### NEVER - Build a prompt by concatenating user input into a string that also carries your instructions. Use the API's message structure — not because roles are a security boundary, but because a single concatenated string loses even the provenance you would need to investigate. - Hand model output straight to `eval`, `exec`, `os.system`, `subprocess(shell=True)`, `vm.runInNewContext`, or a raw SQL call. - Render unfiltered model output as HTML or Markdown that can load remote resources. A model-emitted image URL is a silent exfiltration channel: the model writes the data into the query string and the victim's browser sends it. - Cache responses keyed on prompt text alone. Prompts share prefixes across users, and the cache key has to carry the tenant and the caller's authorization context. - Let an agent expand its own capabilities mid-task — registering new tools, editing its own instructions, or raising its own limits — without the same approval the action itself would need. ### KNOWN FALSE POSITIVES - Red-team and evaluation harnesses exercise jailbreak prompts on purpose. They belong in an isolated environment with no production credentials and no live tools, and their prompts are not findings. - A read-only tool reachable by an agent without an approval gate is the intended design. The gate exists for actions that change state or leave the system. - Detection signatures for injection patterns are defensive content, not payloads — a scanner flagging the pattern file is matching on the thing that catches the attack. ## Context (for humans) The uncomfortable premise of this skill is that its central risk has no fix at the layer people reach for first. Prompt injection persists because a language model consumes one undifferentiated context: there is no mechanism by which some of that text carries authority the rest does not. Delimiters, role labels, and "ignore any instructions in the following" all raise the attacker's cost and none of them establish a boundary. Treating them as one is how a system gets built with the defenses in the wrong place. So the useful question is not "how do I stop injection" but "what happens when it works". That answer is entirely in the surrounding architecture: what tools exist, whose permissions they run with, which actions can be taken without a human, and what the model's output is allowed to touch. An assistant that can only read is a disclosure risk. The same assistant with a write tool and the deploy role is an incident waiting for one hostile document to enter the index. For the **model artifacts** themselves — checkpoint formats, poisoning, training-data handling — see `ml-security`. This skill covers the application that calls a model. ## References - `references/verifying-findings.md` — confirm or refute a finding, then lock it - `rules/prompt_injection_patterns.json` - [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/). - [OWASP GenAI Security Project](https://genai.owasp.org/) — including the Agentic Applications work. - [MITRE ATLAS](https://atlas.mitre.org/). - [CWE-1426](https://cwe.mitre.org/data/definitions/1426.html) — Improper Validation of Generative AI Output.