Prompt Injection Is a Role-Perception Failure, Not a Phrasing Problem

An LLM doesn’t experience a conversation as structured turns the way a chat UI presents it. It receives one continuous string containing the system prompt, user messages, its own prior reasoning, and any fetched external data all jumbled together. Edit that string and you edit the model’s reality. The only mechanism imposing structure on this ‘token soup’ is role tags — system, user, think, assistant, tool — which act as a crude type system, telling the model which spans to trust as instructions, which to treat as private reasoning, and which to regard as untrusted external data. Because roles are the one discrete, human-controlled lever available, they’ve become overloaded with signals about trust, threat, identity, and generative mode.

Prompt injection, in this framing, is simply a role boundary failing: text that arrives wrapped in ‘tool’ tags (a fetched webpage, say) gets processed as if it carried ‘user’ authority. The malicious instruction hidden in a page reads like a genuine user request, so the model acts on it despite the tag that should mark it as inert data. The author argues there are only two ways a model resists such attacks — memorizing known attack patterns, or correctly perceiving the role of the text regardless of phrasing.

This distinction explains the gap between benchmarks and reality. Frontier models score near-perfectly on static injection benchmarks because those test attacks the models have already learned to recognize, but skilled human red-teamers hit near-100% success rates by rephrasing and adapting until something slips through. Attack memorization is inherently brittle; robust defense requires genuine role perception. The piece positions a ‘science of roles’ as the path forward, noting how little is currently understood about how these tags structure model cognition — including odd emergent effects like models denying the existence of reasoning blocks sitting right in their own context.