← All posts

Your AI Doesn't Need to Be Trusted. It Needs a Boundary.

Why the smartest way to use powerful AI is to stop asking whether you can trust it.

By Yacine Kellib

Imagine you hire the most brilliant finance analyst you have ever met.

On day one, she can read every invoice in the company, spot every anomaly, and tell you exactly which suppliers should be paid this month and how much. She is faster than anyone on your team and rarely wrong.

Here is what you still do not do on day one: give her the keys to the bank account.

Not because you distrust her. Because that is not how organizations work. Being brilliant at deciding what should happen has never meant being allowed to make it happen. She can propose a €72,000 payment. Someone with the right role, inside a spending limit, against a valid purchase order, with a second signature above a certain threshold, authorizes it. And the bank's own system carries it out.

Everyone understands this. We built our companies and our software around this distinction decades ago.

Then AI arrived, and we quietly forgot it.

What changed

For years, AI gave us answers. If the answer was wrong, a person read it, shrugged, and moved on. The damage stopped at the screen.

That is no longer the situation. Today's AI agents send emails, edit files, run commands, move money, change infrastructure, and hand tasks to other agents. They can work for an hour without anyone watching. The gap between what the AI thinks and what happens in the world has become very small.

So a mistake that used to produce a bad paragraph can now produce a bad payment. A malicious instruction hidden in a web page that used to produce a bad answer can now produce a leaked customer database.

And the industry's response has been to make the AI more trustworthy: better training, guardrails, filters, monitoring, and a lot of pop-ups asking "are you sure?"

All of that helps. None of it changes one basic fact.

AI is probabilistic, and that will not change

An AI model is not a calculator. It does not follow rules; it makes very good predictions. It can be wrong, misread a situation, be manipulated by something it reads, or run into a combination of circumstances nobody planned for.

Better models make this rarer. They do not make it impossible.

The numbers are public. OpenAI reports that 8.5% of indirect injections still succeed against the most injection-robust model measured, within fifteen attempts, across 1,810 attacks. That is excellent security, and it is still not zero. And the number that matters is the one with fifteen attempts rather than the one with a single attempt, because a serious attacker does not try once (OpenAI, GPT-6 Astra system card, 2026-09-03).

So the question is not how do we make the AI perfect?

The question is how do we make sure that when the AI is wrong, the mistake cannot turn into a consequence on its own?

"Just ask a human" stops working faster than you think

The obvious fix is a person in the loop. Every time the AI wants to do something important, it asks. You click approve or deny.

This works beautifully for ten decisions a day. It collapses at a thousand.

Anthropic has published what happened with its own coding tool, and it is worth reading slowly. Users were approving 93% of the requests the AI made, and the company named the reason: the more prompts people see, the less attention they pay to each one. By August 2026 the approval rate had reached 97%.

Then they ran an experiment. They hired 1,053 professional testers and, part-way through each session, quietly replaced one routine request with a clearly dangerous one. The testers caught it 13.6% of the time. After fifty or more prior prompts, the catch rate fell to roughly 5% (Anthropic, August 2026).

Think about what that means. The safety mechanism was still there. The dialog still appeared. A human still clicked. And the human had, for practical purposes, stopped looking.

I call this Authority Fatigue: when a system asks a person to authorize so many things that the person's approval stops meaning anything.

It is not that people are careless. It is that human attention is finite and the AI's output is not. When the thing you are supervising generates decisions at machine speed, your attention becomes the weakest part of the system, and any attacker knows exactly where to aim.

The tempting shortcut, and why it is only half an answer

If humans cannot review everything, why not let a second AI do the reviewing?

This is exactly what the industry is doing, and it works better than exhausted humans. In that same experiment, an automated reviewer caught 89% of the dangerous commands.

But Anthropic also published the other number: on real-world cases where the AI overstepped, the automated reviewer missed about 17%. Their own description of the failures is telling. The reviewer usually saw the danger. What it got wrong was whether the user had actually agreed to this particular action. It found something that looked like permission and stopped there (Anthropic, March 2026).

That is the same problem, one step to the left. A probabilistic system is now guessing about authority instead of guessing about the task.

A second AI is a useful extra layer. It should not be the final word on what is allowed.

Separate the thinking from the permission

Here is the idea at the heart of ZIFFER, and it is the same idea we already apply to that brilliant analyst.

Let the AI think. Do not let the AI decide what it is allowed to do.

Concretely, an AI system should be split into two parts:

Above the line: intelligence. The model, its memory, its tools, everything it reads and reasons about. This part can be extraordinarily capable. It can also be wrong or manipulated, and we plan for that.

Below the line: authority. A separate system, owned by the organization, that holds the rules: what this agent may do, to which resources, within which limits, under which conditions, and when a human must be involved. It does not think. It checks.

The AI produces a proposal: pay this supplier this amount from this account for this invoice. The authority layer checks that proposal against the rules. Only if it passes does anything execute, and what executes is exactly what was approved, nothing more.

The AI never holds the keys. It never grants itself permission. It never decides that because it concluded something was necessary, it is therefore allowed to do it.

Nothing above the line grants itself authority. That single sentence is the whole architecture.

What this changes for the human

You stop being a button.

Instead of clicking approve a thousand times, you answer a different question once: what should this agent be allowed to do?

It may read customer records for support purposes.
It may update addresses.
It may issue refunds up to €500.
It may never see payment credentials.
It may never export customer data.
Anything over €10,000 needs a second person.

You define the boundary. The system enforces it. The AI works freely inside it. When the AI wants something outside the boundary, that is when a human hears about it, and by then the request is rare enough to deserve real attention.

This is not my invention. It is where the field is heading. NVIDIA's published security model for AI agents puts it in four words: above proposes, below decides (NVIDIA). OpenAI, Anthropic and the US standards body NIST are all building versions of this boundary into their own products and guidance. Anthropic's own conclusion after two years of incidents was that the fixed, rule-based boundary is what saves you when every probabilistic defense misses (Anthropic, May 2026).

The difference ZIFFER argues for is about who owns the boundary. Today each AI vendor builds one for its own product. A company using five models from three vendors inherits five different sets of rules, each written by someone else and changed on someone else's schedule. The boundary should belong to the organization, outlive any particular model, and be something no vendor can widen.

What this does not fix

Honesty matters more than a clean pitch, so here are the limits.

It does not catch a well-formed mistake. If the AI misreads an invoice for €7,200 and proposes €72,000 to a real supplier, within limit, for a real purpose, the boundary will let it through. The proposal is correct in form and wrong in fact. Separation controls what an AI may cause. It does not verify that the AI read the world correctly. That needs a different kind of check, against the original records, and it is a harder problem.

Rules are written by people. A boundary is only as good as the rules inside it, and rules are always incomplete on the day they ship. What separation gives you is that a failure becomes a rule you can inspect and fix, rather than a behavior you can only observe and hope to retrain away.

Small allowed actions can add up. A thousand refunds of €499 each are all under a €500 limit. Budgets and cumulative caps help. The general problem of many individually fine actions producing one collectively bad outcome is not solved by a per-action boundary alone.

The question worth asking

The whole industry is asking: how trustworthy can we make the AI?

That is a good question and the answer keeps improving.

But there is a better one: how powerful can AI become if it never needs to be trusted with authority in the first place?

Your brilliant analyst can be as brilliant as she likes. The bank account has its own rules. That is not distrust. That is how you get to hire the brilliant analyst at all.

The same is true of AI. The safest way to use extraordinarily capable intelligence is not to finally trust it enough to hand it control. It is to design things so that it never needed control to begin with.

Intelligence proposes. Authority decides. Execution verifies. Evidence remembers.

That is ZIFFER.

Sources

← All posts