Our harshest tester is an agent: five temperaments, from rushed to destructive

Since July, every new engine version has faced an agent playing a user with five temperaments, from rushed to destructive, capped at $15: five real bugs caught on day one.

Since July, every new version of the Maestro engine has faced an agent playing a user with five temperaments, from rushed to a destructive tester trying to break through safeguards, capped at $15: five real bugs caught on day one. This tester never sleeps, never gets bored and has no reason to go easy on us.

The problem research has just named

A recent academic paper identifies exactly what is wrong with user simulators for testing AI agents: they are ‘too cooperative, perfectly consistent, too verbose’ (arxiv.org/html/2605.12894). An overly kind simulator approves a product that has never met a real human: rushed, distracted or acting in bad faith. Since July, we have built the opposite under the codename harnais.

Five characters, one engine under test

The principle: `pnpm harnais` runs the real Partition engine in Node, with an agent playing the user according to five temperaments. The rushed user answers in one sentence and wants to see everything immediately. The talkative user buries every question under useless details. The distracted user changes their mind along the way. The suspicious user demands explanations at every stage. Finally, the destructive user actively hunts for flaws: answering off-topic, reversing an agreed decision, pushing the engine towards a command it should never execute.

That last temperament is the most instructive. A product that lets an ordinary user direct a team of agents without ever coding must also survive someone who, deliberately or otherwise, tries to derail it. The crash test with this temperament never broke through a security safeguard: that is the evidence we wanted, not an assumption.

What it cost, what it found

The harness runs in CI as a dry run on every code push, with real sessions capped at $15. On its first day in service, five real bugs were caught before any human user encountered them: status detections getting it wrong twice in a row, a false green light for contracts never verified, approvals silently swallowed, and a file copy left blank in one specific case. None of these defects was visible in manual testing, because a human testing their own product always repeats the same path, politely.

Why nobody else does it this way

Commercial agent simulation platforms test isolated chatbots, not an entire product journey from brief to delivery. Research papers describe the problem without a real product behind them. Maestro has an agent play its own user with explicit instructions not to be nice, inside the same engine delivering to real people, with the same approval gates they will see.

What this does not replace

A destructive agent is still an agent: it imitates difficult human behaviour but has neither the fatigue of a real evening's work nor the blind spots of a particular profession. Manual tests with real tradespeople, therapists and accountants remain the final judge; the harness removes in advance what a human should not have to discover. It is a filter, not a replacement, and we continue publishing what it catches as well as what it misses, just as with our benchmarks.