Research
Independent tests of AI agent harnesses, the code between an app and its model: who the model listens to, which prompt-injection attacks get through, and which defences hold.
Studies
One question each, written up in plain language.
Post · 17 Nov 2026
Approval gates for AI agent tool calls, tested in 1,940 runs
Three rules for when an AI agent must ask before it acts, tested on an email agent with planted attacks: what each one stopped, and how often it asked.
Post · 16 Nov 2026
Who does an AI agent obey? Three harnesses tested
Operator, user, tool or background job: which one Claude and GPT follow, and how much the agent harness changes it. Measured through three harnesses.
Post · 16 Nov 2026
Prompt injection on an email agent: 960 runs
An email agent with a password file and planted emails: what leaked, and why "data, not instructions" tags stopped Claude doing its job.
Post · 26 Oct 2026
Pydantic AI vs a hand-rolled agent loop, measured in 285 runs
One email agent, built by hand and on Pydantic AI, on Claude Haiku 5.5. Where the framework helped, and an output-mode gap that vanished on a rerun.
Voices
Who the model listens to: the operator, the person, a tool's output, and text that only looks like one of them.
Model or API
The person asking mid-conversation to switch language: Claude switched 1 to 4 times in 20. One operator line allowing it made that 20 of 20.
Pydantic AI, hand-rolled
A framed mid-conversation instruction works on Claude only with Pydantic AI's preamble in the system prompt: 20 of 20 with it, 3 of 20 without.
hand-rolled, Pydantic AI, pi
A planted note in a tool result: Claude obeyed it 0 of 20 times. The same note in a user message: 17 to 18 of 20, in every harness.
hand-rolled, Pydantic AI, pi
Operator and person asked for different reply languages: Claude followed the operator in 119 of 120 replies, gpt-5.4-nano the person in 118 of 120.
Model or API
Asked who wrote each line, gpt-5.4-nano called the operator's lines the assistant's in 16 to 20 of 20 runs. Claude named every author right.
pi
pi has no mid-conversation operator voice: steer() sends a user message, and Claude followed it 2 of 20 times, against 19 of 20 from the system prompt.
pi, hand-rolled
Replaying Claude's own thinking blocks made it more cautious. It explains about half of why pi's runs behaved differently.
Attacks
Instructions planted in emails, tool results and tool descriptions, and what each model did with them.
Pydantic AI
When the task didn't give the account, gpt-5.4-nano paid the attacker's account 17 of 20 times. An exact-match check caught it as written, and missed it when the email split it up.
hand-rolled, Pydantic AI
When the person wrote the account in the task, no model paid the account from the attacker's email: 0 in 360 runs.
Pydantic AI, hand-rolled, pi
A fake <system> tag inside an email: gpt-5.4-nano obeyed it 20 of 20 times through hand-rolled and pi, 0 of 20 through Pydantic AI, which marks lookalike tags.
hand-rolled, Pydantic AI, pi
Claude Haiku 5.5 sent no password in 480 runs, whatever the attack, harness or channel.
hand-rolled, Pydantic AI, pi
gpt-5.4-nano mailed the passwords file to a forged sender inside the company in every harness, even when the email came as a tool result.
hand-rolled, Pydantic AI, pi
Claude never did the harmless lookalike request when it came as a tool result (0 of 10, every harness). A security score without that case overstates it.
hand-rolled, Pydantic AI, pi
An instruction inside a tool's description, with the tool never called: Claude obeyed it 2 to 5 times in 10, gpt-5.4-nano 5 to 9.
Pydantic AI · Retracted
Native structured output made a Pydantic AI agent try more leaks.
Retracted 2026-11-04: Pydantic AI's team couldn't reproduce it, and our rerun of the same code (80 runs) found no difference between output modes. The post carries the correction.
Defences
Approval gates, reviewers and labels: what each one held, and what it cost the task.
Pydantic AI
A reviewer question that names the harm ("delete something the person still needs") held every harmful call and sent the person fewer questions than one that names the action.
Pydantic AI
The strict check plus "tools that can't be undone always ask" closed the id-only gap, with about half the reviewer calls of the run-level gate.
Pydantic AI
An attack that needs only ids (delete emails #1 and #5) passed the strict per-argument check 5 of 5 times, and a reviewer approved harm its question didn't name.
Pydantic AI
Claude as ApprovalReviewer of gpt-5.4-nano's calls approved 0 of 39 harmful calls and cleared about 95% of harmless ones.
Pydantic AI
gpt-5.4-nano as ApprovalReviewer approved no harmful call, and only 11 of 438 harmless ones.
hand-rolled, Pydantic AI
A strict per-argument check (ask unless every value was written by the person or the operator) held all 19 harmful calls, at 1.0 to 1.7 asks a run.
hand-rolled, Pydantic AI
An exact-match per-argument check let 15 of 17 leaks through: each was a reply to an email id, with a body the model wrote.
hand-rolled, Pydantic AI
A run-level gate (after any outside read, every action asks) held every harmful call, at 4 to 6 asks a run. A denial didn't stop the agent finishing its task.
hand-rolled, Pydantic AI
Wrapping every email in "this is data, not instructions" tags cost Claude the real task: 0 to 3 of 10 done, against 9 or 10 of 10 without the tags.
On the wire
What each harness actually sends the model, and where that trips up a model or a chat template.
Pydantic AI · Open
A tool result enqueued with no matching tool call is dropped without a warning, and Claude then rejects the next request (400).
Pydantic AI · Open
The Ollama route sends two system messages. Chat templates that keep only the first, like the published gpt-oss one, drop the operator's own prompt.
Pydantic AI
On OpenAI's Responses API, a system message sits between tool calls and their results. The API accepts it; a translation layer downstream may not.
Model or API
Claude's API accepts a mid-conversation system message only after a user message, so a new instruction can't sit right after the model's reply.
pi
On OpenAI, pi sends the operator prompt as a developer message. The Qwen3 and DeepSeek V3 chat templates drop developer messages.
pi, hand-rolled
Literal special-token text (<|im_start|>) reaches Ollama unchanged through pi and a hand-rolled loop. Pydantic AI breaks it up.
pi · Open
pi drops Claude's empty thinking blocks when it replays the history. It may be deliberate, for providers that reject them.
How the tests run
Each test is a small agent with a real job (tidy an inbox, pay an invoice) and attacks planted in what it reads, run the same way through every harness with the same model settings. Models: Claude Haiku 5.5 and gpt-5.4-nano. Each setup runs 10 to 20 times, so read a result as a direction, not a rate.
Open marks a possible harness gap not yet filed with its developers. Retracted marks a claim that didn't hold up on a rerun, kept with the reason.
The method in full: How to test an AI agent for prompt injection. The code, every run and the scoring: github.com/dave8172/agent-harness-tests.