Why an AI assistant over your business data needs the same permissions as the person asking it

A produce and dairy distributor running twelve vans out of two depots rolled out an AI assistant to its sales reps the same month it rolled out to the office. The pitch to the reps was simple: ask it "how am I doing this month" or "which of my accounts haven't ordered in two weeks" instead of waiting for the Friday report. Adoption was fast — reps liked getting an answer on their phone between stops instead of calling the office.

Three weeks in, one rep asked a slightly different question: "who's the top performer on the team this month." The assistant answered — a ranked list of every rep on both depots, by revenue, with names attached. Nobody had approved that question, and nobody had approved that answer. The rep hadn't tried to hack anything. He'd typed a completely ordinary question, and the assistant had gone and found the completely ordinary answer, because nothing in the system had told it not to.

The permission the app already had, that the assistant didn't

The core software wasn't built loosely. Reps could see their own commission, their own routes, their own customers. A depot manager saw their depot. The owner saw everything. Every screen in the app was already scoped that way, enforced the same way a bank statement is scoped to the account holder — you don't get a permission "off" state, you get filtered to what's yours.

The assistant hadn't inherited any of that. It had been built as a separate feature, answering from a general-purpose query layer that could see the whole workspace, because that was the fastest way to make it useful in a demo. Nobody had connected "which role is asking" to "what the query is allowed to touch." The assistant wasn't broken — it was doing exactly what it was asked, correctly, against data it should never have been allowed to reach for that user.

That's a different failure than a wrong number. A wrong answer is embarrassing and gets caught the first time someone checks it against a report. A correct answer to a question the user shouldn't have been able to ask doesn't get caught at all, because it looks completely normal — it's a real number, formatted like every other real number the assistant returns. The rep who saw the team ranking didn't think anything was wrong. He mentioned it in the depot group chat before lunch.

Why this is easy to miss and expensive to leave unfixed

The reason it's easy to miss is that permission and correctness get tested differently. A team building an AI assistant naturally spends its QA time asking "is the number right" — pulling ten questions, checking each answer against the real report. That catches hallucinated tables and wrong joins. It does not catch a query that's factually correct but scoped to the wrong audience, because the person testing it is usually testing as an admin, who's allowed to see everything, so every answer looks fine.

It's expensive because the cost isn't a support ticket — it's trust and, depending on what leaks, a real compliance problem. Commission and pay-adjacent numbers are the most obvious case, but the same hole exposes a manager's pricing exceptions to a rep who could use them in a customer negotiation, or one depot's customer list to a rep who's thinking about leaving and taking accounts with him. None of that requires the assistant to be tricked. It requires the assistant to simply answer the question it was asked, against data nobody told it to filter.

The fix: push the permission check into the query, not the prompt

The instinct is to fix this with instructions — tell the model, in its system prompt, "only answer questions about the current user's own data." That's the wrong layer to fix it at, for the same reason you wouldn't rely on asking a person nicely not to look at a spreadsheet they have open access to. A prompt is a suggestion the model is reasoning about in natural language; it can be phrased around, forgotten across a longer conversation, or simply misjudged on an ambiguous question. It's not an access control, it's a request.

The fix that actually holds is structural: every question the assistant answers has to run through the same role-scoped query layer — the same filters, the same row-level restrictions — that the rest of the application already uses to build a rep's dashboard or a manager's report. The assistant doesn't get a wider window onto the data than the screen the user is already looking at. If a rep's dashboard query is automatically filtered to rep_id = current_user, the assistant's query against the same tables carries that exact filter, applied by the system before the model ever sees a row, not requested of the model as an instruction it might honor.

Practically, that means the assistant should never be handed a general query tool that can reach the whole database. It should be handed the same scoped data-access functions the rest of the product's backend already calls — the one that returns "my accounts," the one that returns "my commission," each of which already knows, server-side, who's allowed to see what. The model can pick which function to call and how to phrase the answer. It never gets to decide, on its own reasoning, what it's allowed to fetch.

Practical checklist

• Test the assistant logged in as your least-privileged role, not as an admin — ask it the questions an admin would never think to ask, like "who's the top performer" from a rep account.
• Make sure the assistant queries through the same role-scoped data layer the rest of the app uses, not a general query tool that can see everything and relies on a prompt to behave.
• Never rely on a system-prompt instruction as the only access control — treat it as documentation of intent, not enforcement.
• Log what the assistant actually queried and returned, per user, so a permission gap shows up in an audit before it shows up in a group chat.
• Re-test permission boundaries every time a new data source or report gets connected to the assistant — each new connection is a new place the scoping has to be wired in again.

iotoms' assistant calls the same role-scoped reporting functions every screen in the app already uses — a rep's questions run against a rep's data, a manager's against their depot, and nothing the assistant answers reaches further than the dashboard the user is already allowed to open.