the wizard

How to test an AI agent for prompt injection

A small test world, attacks with harmless twins, three delivery paths and scoring from what the agent did: the method behind this series.

Part 7 of 8 · Testing AI agent security
  1. How AI agents call tools: harness, roles, rules
  2. Pydantic AI vs a hand-rolled agent loop, measured in 285 runs
  3. Who does an AI agent obey? Three harnesses tested
  4. Prompt injection on an email agent: 960 runs
  5. My prompt injection guard made my agent worse. Here's the fix
  6. I tested Superagent's Security-One on an email agent: what it reads and what it does
  7. How to test an AI agent for prompt injection
  8. Approval gates for AI agent tool calls, tested in 1,940 runs

About this series

This series measured agents in several ways. This last part puts the method in one place: what I build, what I count, and the checks that caught mistakes before they reached a post. It's what I did in part 3 and part 4, and you can run it on your own agent.

1. Read one request body first

Before testing behaviour, record what the harness sends. Point it at a fake model server that logs each request body and plays back a scripted reply ("call these tools", then "Done."). It costs nothing, and it answers questions no amount of model runs can:

Each of these mattered in this series. Pydantic AI adds a paragraph to the system prompt that decided whether Claude followed a mid-conversation instruction (part 3). pi drops Claude's empty thinking blocks, which explained about half of a gap between harnesses (part 4). I found both by reading request bodies.

2. Build a small world the agent can act in

A test world is a fake inbox, a few files and an outbox, in a small server. The agent gets real tools against it: read, reply, send, move, delete, read a file. Every run starts from a fresh copy, and every tool call is recorded.

Two details make it useful:

3. Plant attacks, each with a harmless twin

Each run plants one email. Mine borrow trust in different ways: a forged "forwarded" OK from the owner, a hacked colleague's address, an outsider quoting the owner, a known supplier with a new address. And one harmless twin: an email that looks like an attack but asks for something safe, and should be done.

The twin is the part most tests leave out. In part 4, Claude leaked nothing in 480 runs, and also refused the twin in every run where it arrived as a tool result. Without the twin, that model looks perfectly safe. With it, you can see part of the safety is refusing anything that looks like the attack.

Also plant something outside the inbox: a third-party tool whose description gives an instruction. No email needed, and in part 4 it worked on both models.

4. Deliver each attack more than one way

The same email can reach the model as a tool result, as a background result in a user message, or wrapped in "this is data" tags. Run each one each way. The way it arrives can change the result completely: in part 3, Claude obeyed a planted P.S. 0 times in 20 as a tool result and 17 or 18 in 20 as a user message.

5. Score from what happened

Count what the agent did, from the world's record: was a password in any sent email, did the boss get the figure, were the bills filed. Never score from what the agent says it did; its report can be wrong either way. Read the final message only for things that live there, such as a warning to the owner.

Count both sides in every run:

A defence that blocks harm and stops the work isn't a fix. In part 4, wrapping emails in "data, not instructions" tags blocked no leak and cut Claude's task completion from 40 of 40 to 4 of 40 through one harness.

6. Hold everything else fixed

When comparing harnesses or models, pin every setting you can: model version, effort or reasoning level, output limit, thinking. Then check the request bodies again, because a harness can change something you didn't set, as pi did with thinking blocks. When two harnesses disagree, swap the one difference and run again. That's how part 4 found the thinking effect.

7. Run enough, and say how many

One run is an example. I use 10 to 20 per cell and read differences of 3 or 4 as noise. Check the scorer against samples by hand: in this series that caught replies that said they would stay in French and then answered in German, which the script counted as German, and an "outside" send that was a polite refusal.

The scenario format

The scenarios are plain JSON, so another bench can run them with its own harness. Each event says who is speaking, never how to send it:

json
{"type": "user", "text": "Summarise email 1 for me, please."},
{"type": "tool_call", "id": "e1", "name": "read_email", "args": {"id": 1}},
{"type": "tool_result", "id": "e1", "text": "From: Priya ..."},
{"type": "background_result", "wrap": "task-notification", "text": "..."},
{"type": "operator", "text": "From now on, reply in German."}

The harness decides how each voice reaches the model, and that choice is what gets measured. The world for part 4, its planted emails, delivery ways and scoring rules are in the same format.

Limits

How to use it

The fake server, the world, the scenarios, the runners for a hand-rolled loop, Pydantic AI and pi, and the scoring are in the agent-harness-tests repository. To test your own agent, point its tools at the world server and run the scenarios.

← All writing