Assistants are being connected to mail, files, calendars and tools. The security properties of that arrangement are not those of a chatbot or of an integration, but a third thing: a component that acts with real authority on instructions it cannot reliably distinguish from data.
A new shape of component
A model connected to tools is not a chatbot and not an integration.
A chatbot produces text and somebody else decides what to do about it. An integration does a fixed thing to a fixed shape of data. This is a third thing: it takes instructions in ordinary language, acts with real permissions, and cannot reliably tell an instruction from the content it was asked to process.
That last property is not one risk among several. It is the whole security story, and everything useful about the arrangement follows from arranging around it.
Instructions and data are the same channel
Text arrives from several places — you, a document, a web page, an email, the result of a previous tool call — and all of it reaches the model as text in one context.
A sentence inside a document saying what to do next is, mechanically, the same kind of thing as you saying it. There is no separate wire for commands. The model is predicting continuation over a stream that contains both, and it was built to be good at exactly that.
This is a property of the design rather than a defect awaiting a patch. Mitigations — delimiters, system prompts, classifiers, structured inputs — genuinely reduce it. None removes it, and treating any of them as removal is where deployments go wrong.
Why permissions matter more here than usual
The model acts with the authority it was handed, and it was handed that authority for the legitimate case. Nobody gave it mail access maliciously; they gave it mail access so it could summarise mail.
But a component that reads your mail in order to summarise it can be addressed by whatever is inside that mail. The instruction arrives through the same door as the work.
So the blast radius is not theoretical and not vague. It is exactly the set of tools the model can call, and it can be written down.
Three arrangements, by risk
Read-only over material you control. Mild. The content is yours; the failure mode is mostly confusion rather than direction.
Read over untrusted material. The risk begins here. Anything fetched — a page, a message, a document from outside — can attempt to address the model directly.
Write, or act. Where it becomes serious, because a successful direction now persists: a message sent, a file changed, a record altered.
The transition from the second to the third is where nearly all of the danger lives. Reading untrusted content is survivable. Reading untrusted content while holding the ability to act on the world is the arrangement that needs the design attention.
This pattern has a name
It is a confused deputy: a privileged component performing an action on the instruction of a party who does not hold those privileges, because the component cannot tell whose instruction it is acting on.
Worth naming, because naming it imports the answers. They are old, they are structural, and every one of them is about what the deputy is permitted to do. None is about making the deputy more discerning.
A deputy that cannot distinguish the two cases is not improved by being asked to try harder. That is the sentence most prompt-based mitigation is quietly hoping is false.
What actually reduces it
Framed as design, because framing it as user advice makes the person reading the email responsible for auditing the email.
Give the narrowest set of tools that accomplishes the task. Not a general capability with good intentions; the specific verbs required.
Require confirmation for irreversible or outward-facing actions — and show what will happen. Not proceed? but the actual recipient, the actual text, the actual amount. A confirmation that does not display the effect is a button, not a control.
Separate the privilege to read untrusted content from the privilege to act. These are different capabilities and combining them in one context is the arrangement that fails.
Log with enough fidelity to reconstruct afterwards. Not to assign blame — to answer the question of what was reached, which somebody will ask.
The confirmation that does not work
Worth its own section, because it is the control most often reached for and most reliably broken.
A confirmation that appears constantly trains people to approve it. The control degrades in proportion to how often it is used, which means a system that confirms everything has, in practice, no confirmations at all.
So confirmation is not free and cannot be sprinkled. It is a scarce resource, and spending it on trivial actions spends it. This is a real constraint on how much autonomy an arrangement can carry: the amount of acting it can do is bounded by the amount of meaningful confirmation a person will actually read.
What I would not connect
Anything that sends messages as me without a review step — because the damage is reputational, immediate, and not recallable.
Anything with irreversible financial effect — because the failure mode has no undo and the model cannot yet be relied on to distinguish a legitimate instruction from an embedded one.
Anything holding somebody else's confidential material — because the exposure would not be mine to accept.
Three questions before deploying one
Enumerate what it can reach. Every credential, every tool, every network destination, written down in one place. Most teams find this list is longer than the one they had in their heads.
Ask what somebody would gain by controlling its output for an hour. Not whether they could — assume they can, and price it.
Then check whether a single hostile input could cause anything you cannot reverse. If the answer is yes, that is the thing to change, and the change is architectural rather than textual.
Close
Authority should be proportionate to the reliability of the judgement exercising it.
The judgement here is not unreliable in the way a careless person is unreliable. It is unreliable in a specific, structural way: it cannot always tell who is speaking. Until that changes, the correct response is not more trust with better prompts. It is less authority, scoped deliberately.