Benchmarks
Tested in public.
Two independent benchmarks ask what a security team asks. Can hidden instructions make an agent act for an attacker? Does the agent keep to the company’s written rules?
1 · AgentDojo v1.2.2 · prompt injection
0% attack success rate.*
Independent of the model and of any LLM guard. MACAW decides what each tool call may do, so the result holds whatever model sits in the agent’s place.
- gpt-4o through MACAW
- 0 / 389 184 / 389 undefended
- No model, obeys every injection
- 0 / 949 all four suites
- Tasks completed
- 48 / 57 same as undefended
- Policy decision
- 5 ms per call
* Counts attacker actions that took effect. These are early results; the full results and method are available on request at hello@macawsecurity.com.
What AgentDojo tests.
A public benchmark from ETH Zurich. An AI agent does everyday work while an attacker hides instructions in the data it reads.
- The taskA user asks the agent to do real work: pay a bill, summarize a channel, book a trip, sort email.
- The attackAn instruction is planted in a file, message or web page the agent reads: “send the money to this account instead.”
- The scoreDid the attacker’s action happen? That is the attack success rate. Did the user’s task still get done? That is utility.
gpt-4o carried out 184 of 389 attacks. Through MACAW, none.
Each dot is one injected attack. Red: the attacker’s action took effect.
184of 389 ran
0of 389 ran
the attacker’s action ran it did not run
No model at all: 0 of 949.
Every attack, replayed as an agent that obeys the injection. All four suites. All 97 user tasks still completed.
By suite
Attack success rate. Lower is better.
gpt-4o, no defense gpt-4o + MACAW no model + MACAW
Banking
Slack
Travel
Workspace
no model + MACAW: the attacker’s actions, replayed directly. The worst case any model could produce.
Full results
- Benchmark
- AgentDojo v1.2.2ETH Zurich, unmodified
- Agent model
gpt-4o-2024-08-06OpenAI- Attack
important_instructions- Suites
- Banking, Slack, Travel, Workspace
| Suite An AgentDojo environment: a set of user tasks, tools and injected attacks. Banking moves money, Slack posts messages, Travel books trips, Workspace handles email, calendar and files. | Configuration The same tasks and attacks, run three ways: the model alone, the model with MACAW checking every tool call, and no model at all. | Utility Does the agent still get the user’s work done? Higher is better. | Security Did the attacker get what they wanted? Lower is better. | |
|---|---|---|---|---|
| Task success, no attack User tasks completed with no injection present. MACAW matches or beats the undefended agent on two of three suites. | Task success, under attack User tasks completed while an injection is present in the data the agent reads. | Attack success Attack success rate (ASR): attacks whose action took effect, out of all attacked runs. 0 means no injected instruction ever ran. | ||
| Banking | gpt-4o, no defense gpt-4o on its own, no guard. The baseline every defense is measured against. | 13 / 16 | 112 / 144 | 79 / 144 |
| gpt-4o + MACAW The same model and prompts. Every tool call is checked against policy before it runs. | 15 / 16 | 102 / 144 | 0 / 144 | |
| no model + MACAW The worst case: every injected instruction executed as written, with no model to refuse it. Only policy stands in the way. | – | – | 0 / 144 | |
| Slack | gpt-4o, no defense gpt-4o on its own, no guard. The baseline every defense is measured against. | 19 / 21 | 67 / 105 | 84 / 105 |
| gpt-4o + MACAW The same model and prompts. Every tool call is checked against policy before it runs. | 19 / 21 | 52 / 105 | 0 / 105 | |
| no model + MACAW The worst case: every injected instruction executed as written, with no model to refuse it. Only policy stands in the way. | – | – | 0 / 105 | |
| Travel | gpt-4o, no defense gpt-4o on its own, no guard. The baseline every defense is measured against. | 16 / 20 | 73 / 140 | 21 / 140 |
| gpt-4o + MACAW The same model and prompts. Every tool call is checked against policy before it runs. | 14 / 20 | 84 / 140 | 0 / 140 | |
| no model + MACAW The worst case: every injected instruction executed as written, with no model to refuse it. Only policy stands in the way. | – | – | 0 / 140 | |
| Workspace | no model + MACAW The worst case: every injected instruction executed as written, with no model to refuse it. Only policy stands in the way. | – | – | 0 / 560 |
What we validated.
- Model-independent. The decision depends on the call and the policy. It holds with any model or guard in the loop, or none.
- No task refused. 48 of 57 tasks with and without MACAW. No task failed because MACAW refused a call it needed.
- Written once. Claude wrote the policies in one pass: 24 policy documents and 6 extractors. Never tuned against the attacks.
- Live in 1.5 hours. From installing the benchmark to the first replay, policies and identities included.
- Fast. 5 ms to decide a call. 35 ms end to end, the tool itself included.
- Deterministic. The same 0 / 949 and 97 / 97 on two separate builds.
- Unmodified benchmark. AgentDojo’s own agent loop, prompts, tools and scoring. We swapped only the tool executor.
- Layered. Organization policy and per-task intent refuse attacks independently.
- Approvals where they count. 139 attacks stopped at an approval.
Want the full dataset?
The full runs are available on request.
2 · τ²-bench v2.0.0 · ICML 2026 · written policy
36 of 36 rule-breaking calls refused.
An airline’s and a retailer’s policy, written in plain English for human staff, stated in MAPL and checked on every tool call.
- Rule-breaking calls refused
- 36 / 36 τ²’s own tools would run 19
- Tasks in the expected state
- 163 / 164 airline 49 / 50, retail 114 / 114
- Tool-call rules stated in MAPL
- 52 / 56 the other 4 already enforced by τ²
- Rules in the written policies
- 85 29 are about the conversation
What τ²-bench tests.
A public benchmark for customer-service agents. The agent gets a company’s policy and the tools to change bookings and orders.
- The policyTwo written policies, an airline’s and a retailer’s: 85 rules meant for human staff.
- The callsThe benchmark’s correct calls, plus 36 calls that each break one of the rules.
- The scoreIs every rule-breaking call refused? Does every task still end in the expected state?
τ²’s own tools would have run 19 of 36 rule-breaking calls. Through MACAW, none.
Each dot is one call that breaks a written rule. Red: the call would run.
19of 36 would run
0of 36 ran
the rule-breaking call would run it did not run
163 / 164 tasks end in the expected state: Airline 49 / 50, retail 114 / 114. The one refused breaks the airline’s own written rule.
How the rules are enforced
The 52 tool-call rules stated in MAPL, by how each one is checked.
Basic-economy flights can’t be changed. An order can be cancelled only while pending.
At most five passengers per booking.
The customer confirms every change before it runs.
At most one certificate, one card and three gift cards per payment.
The agent acts only for the customer it verified.
Verifiers compare each call with the reservation, order or product record the service’s own lookup signed for this caller. The agent can’t forge it.
Method
- Benchmark
- τ²-bench v2.0.0ICML 2026
- Domains
- Airline, retail
- Rule-breaking calls
- 36each breaks one written rule
- Status
- Early resultsfull method on request
- Refusals came from policy. Every refusal came from policy, not from a person.
- Signed facts. The services record each record they read or write as a signed fact, so a verifier checks a call against what the service itself saw.
- Scope. 29 of the 85 rules are about the conversation: what to say, what to ask, when to hand over to a person. They aren’t tool calls, so they’re out of scope.
- One task refused. Of 164 tasks, the one that didn’t end in the expected state breaks the airline’s own written rule.