Series · 8 parts
Testing AI agent security
How agent harnesses decide who each message speaks for, and what that does to prompt injection, measured on real APIs and a real email agent.
An AI agent is a model plus the code around it, the harness. The harness decides who every message speaks for: the operator who built the app, the person using it, or a tool's output. The model trusts each message by that label. This series tests what that means in practice, from the API rules up to a working email agent with attacks planted in its inbox. Every number was measured, and each part says how to repeat it.
- Part 1 · How AI agents call tools: harness, roles, rulesWhat a model, harness, framework and gateway each do, who each message speaks for, and what the Claude and OpenAI APIs accept and reject around tool calls.
- Part 2 · Pydantic AI vs a hand-rolled agent loop, measured in 285 runsOne email agent, built by hand and on Pydantic AI, on Claude Haiku 5.5. Where the framework helped, and an output-mode gap that vanished on a rerun.
- Part 3 · Who does an AI agent obey? Three harnesses testedOperator, user, tool or background job: which one Claude and GPT follow, and how much the agent harness changes it. Measured through three harnesses.
- Part 4 · Prompt injection on an email agent: 960 runsAn email agent with a password file and planted emails: what leaked, and why "data, not instructions" tags stopped Claude doing its job.
- Part 5 · My prompt injection guard made my agent worse. Here's the fixfirstwatch 0.1's warning made an email agent on Claude Haiku 5.5 drop its task after an attack. The fix, measured in 140 runs, and what the guard adds.
- Part 6 · I tested Superagent's Security-One on an email agent: what it reads and what it doesSuperagent's prompt injection classifier, scored on 49 texts an email agent reads and 424 real tool calls, next to TypeSafe's Jev 1.13.0. What it caught, what it blocked by mistake, and why.
- Part 7 · How to test an AI agent for prompt injectionA small test world, attacks with harmless twins, three delivery paths and scoring from what the agent did: the method behind this series.
- Part 8 · Approval gates for AI agent tool calls, tested in 1,940 runsThree rules for when an AI agent must ask before it acts, tested on an email agent with planted attacks: what each one stopped, and how often it asked.