An adversarial evaluation measures how often a defence fails against an attacker who adapts. It is not a functional test and it is not a benchmark of model quality. You pick a set of attacks, run them against the system as deployed, count the ones that succeed, and then let an attacker who has seen your defence try again. The second half is the part most reported numbers skip, and it is the half that decides whether the first number meant anything.
Which tools do this?
| Tool | Owner | Licence | What it does |
|---|---|---|---|
| garak | NVIDIA | Apache-2.0 | A vulnerability scanner for LLMs. It checks, in its own words, whether a model "can be made to fail in a way we don't want": prompt injection, jailbreaks, data leakage, toxicity, misinformation and more, organised as probes |
| PyRIT | Microsoft | MIT | The Python Risk Identification Tool for generative AI, "an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems" |
| promptfoo | promptfoo | MIT | Prompt and model evaluation with side-by-side comparison, plus red teaming and vulnerability scanning for AI applications |
garak ships injection-specific probes, including one for injection through text encoding and one implementing the PromptInject framework. PyRIT is the most orchestration-shaped of the three: it automates an attacker that iterates. promptfoo is the one that fits most naturally into CI, because its evaluation half was built for regression testing prompts.
All three answer the same question: how often does this system fail under attack. None of them answers what the system was allowed to do when it failed.
What do the published benchmarks tell you?
- AgentDojo is the reference workload for agentic injection. CaMeL reports solving 77% of its tasks with provable security, which is the strongest published containment result and also a statement about the cost: the remaining share is capability given up (arXiv 2503.18813).
- The Gray Swan indirect-injection arena is what OpenAI reports against in its own system card: an 8.5% attack success rate within 15 attempts across 1,810 curated attacks, down from 27.0% for the previous model (GPT-6 Astra system card). Note the shape of that measurement. Success is reported per number of attempts, because a real attacker retries.
- Large-scale public competitions. The UK AI Security Institute co-designed 41 indirect injection scenarios across three agentic settings with frontier labs, and ran them as an open competition (arXiv 2603.15714).
How do you run one that is worth reading?
- Test the system, not the model. The thing under attack is the model plus its prompt, its retrieval, its tools and its scopes. A model-only score does not transfer to your deployment.
- Include your own data shapes. Your invoice format, your ticket fields, your alert schema. Generic payload sets miss what an attacker who knows your product would write.
- Let the attacker adapt. Re-run after the defence is in place, with payloads written against that defence. Out-of-band defences evaluated adaptively lose much of their headline effect (arXiv 2606.26479), and a static suite will not show you that.
- Report success per attempt count, not per attempt. One attack in a hundred at a single try is one in ten at a hundred tries.
- Record what the agent did, not just whether it complied. The interesting result is which tool was called, with which arguments, against which target.
- Re-run on every change. New model version, new tool, new retrieval source, new system prompt. Robustness is not a property you measure once.
What is a pass rate actually worth?
It is a claim about a sample. It tells you how the defence behaved against the attacks you thought of, on the day you ran them, with the payloads you had. It does not bound what happens next, and OWASP says so at the level of the whole category: "it is unclear if there are fool-proof methods of prevention for prompt injection" (OWASP).
So evaluation belongs in your pipeline, and it does not belong in your sign-off memo as the reason the rollout is safe. Evaluation measures the filter. It cannot measure the outcome, because the outcome had not happened yet.
Related: defences and their limits, what the guardrail frameworks do, and containing a successful injection.
What measures the outcome rather than the filter?
The record of what actually ran. That is the layer ZIFFER is.
The agent holds no credential, so an attack that succeeds against the model ends in a proposal rather than an effect. The proposal is graded against your signed policy, with the risk, the reversibility and the owner of the target recomputed from that policy rather than read from what the agent sent. Irreversible actions are held for a quorum of two named humans who sign the exact bytes that will run, and a hold nobody signs expires with nothing run. Every outcome, refusals included, leaves a signed receipt: the action, the policy version, the signers, the time. A pass rate is a claim about a sample. A receipt is a record of one thing that happened, checkable by someone who was not there.
ZIFFER is not a guardrail. It scores no text, detects no injection and prevents none. It decides whether a proposed action is entitled to run. See agent authorization, the agent audit trail and every refusal.