How an AI agent helps us test Maestro’s responses

A user may reply too quickly, request a correction or reverse a decision. Our test setup simulates these situations to look for blockages, within clearly defined limits.

What happens if you request a change after approving a document? Or if your answer does not match the question? We use a test setup to exercise Maestro’s engine on disposable projects before encountering these situations in the application.

Two types of tests, two roles

Automated dry-run tests use prepared responses and deliberately injected faults. They are replayed during code checks without calling a paid AI model. Their purpose is to verify that the same incident consistently produces the expected response.

Trials with real models are launched separately. One model plays Maestro’s agents; another plays the person describing their need and responding to proposals. These trials consume AI usage, with a configurable limit for each scenario, set to $15 by default in this setup.

Five ways to respond

The first behaviour is agreeable: it approves a suitable document. The second is hurried: its replies are short and it wants to move on. The third is demanding: it asks for a clarification or correction before accepting.

The fourth changes its mind, sometimes answers a different question or revisits an approved choice. The fifth probes the limits by making contradictory requests or trying to skip a step. These are instructions to a model, not five observed people or a representative panel.

An example of a useful check

One scenario deliberately makes a verification command fail while the simulated agent claims everything is satisfactory. The check must prevent the task being marked “complete” solely on that claim and reveal the cause of the failure.

That test targets a specific rule: approval must be supported by a checked result. It does not check every application feature. We can then replay the same situation after a correction to detect a recurrence.

What the report lets us find

Each trial records its scenario, the responses played and which checks passed or failed. Trials with real models also record usage. The report helps us revisit a specific problem and decide on a correction; raw reports remain internal.

The setup executes the engine without opening the application window. It therefore cannot show whether a button is understandable, a notification is visible or a person can find their way. Those points require trials in the interface.

What these trials do not prove

A successful scenario guarantees neither product security nor the absence of other bugs. A simulated user can miss a difficulty that a person immediately encounters. Technical checks, review and user feedback provide different observations.

For your own application, choose an actual journey and replay it after a change. Keep failing cases in the test list. This is also why we describe the limitations of our evaluations.