Assume the Model Gets Fooled

The 2026 OWASP LLM Top 10

The just-released OWASP Top 10 for LLM Applications includes some interesting changes. It opens with a sentence that frames the rest of the document:

Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks.

That’s a new stance, one overdue in hindsight, because it follows from a very specific technical fact. A language model reads its system prompt, the user’s question, a retrieved document, and a tool’s output as one token stream. Current architecture doesn’t separate an instruction from data. That’s just how it’s built.

So that means that you don’t close prompt injection in the same way as SQL injection. You need to limit or bound what a successful injection can reach. There are three general boundaries that do work.

The first is what tools the model can call. On its own, a model produces text. It can’t read a file, send an email, or query a database until a developer gives it a function to call or invoke. So a list of registered tools is essentially the complete set of actions that a model can take. Which means that anything not on the list is out of the reach of an agent.

So the tool list is essentially an authorization decision, and falls into CISSP domain 5. Here, the agent is the subject, and the tools are its permission set. The traditional rule is to grant the minimum necessary to do the job.

What changes is how much weight that rule carries. Since the agent can be talked into doing things, whatever it’s permitted to do, anyone who can get text in front of it is effectively now permissioned. Least privilege stops being hygiene to follow, and becomes the only thing standing between a successful injection and what is within the agent’s reach.

That’s the “confused deputy” problem, named by Norm Hardy in 1988, after a compiler that would overwrite the system’s billing file. The user didn’t have permissions to touch that file, but the compiler did. And the compiler had no way to tell its own business from a file a user had handed it. The user passed a filename. A filename indicates which file, not who’s allowed to touch it. So the compiler had to supply the authority itself, and its own was the only authority it had. Hardy’s fix was to make the deputy act with the caller’s authority, and OWASP arrived at the same place.

Their mitigation is to execute tools in the user’s context, and preserve that context across chained tools and agent calls, never falling back on the agent’s own broader service account. Essentially, the agent should use the user’s permissions, and it should ensure the next agent uses that same permission set.

Note that deciding which tools should be used, and how much authority each has to operate, are important and separate decisions. Reducing the number of tools but leaving their elevated permission scope doesn’t reduce the risk.

What the agent remembers between sessions is important. Agents can carry long-term memory forward between sessions via RAG or vector store, and these memories get pulled back into the context window each session and read as trustworthy. Because the model has no way of telling a memory it formed from one that someone else planted, one poisoned entry can be used in a subsequent session.

This idea relates to traditional data storage handling as defined in CISSP Domain 2. In this case, a memory needs to be classified with a retention limit, and controls placed over who can access and modify it. Typically in this context, a memory is treated like a product feature, which is why it ends up as an asset in your environment with no owner or associated retention policy.