Why we also describe failures in our AI agent tests

A good score can hide a missing feature. Two results from our internal trials show why we need to examine errors and specify what we actually measured.

One time-tracking application received 92.5 out of 100 in our internal evaluation. A later check revealed that it would not compile and erased its database on startup. That case forced us to reconsider how we judge a result, even one with a high score.

The first defeat concerned a specific need

In the first comparison on habit tracking, our method scored 82.1 against 90.8 for BMAD-METHOD v6. This measured design coverage: the judges checked whether the expected rules appeared in the documents. It was not a software reliability percentage.

The comments highlighted dates, duplicates and missing rules. They helped us prepare the next trials. We retain that result in our account of the campaigns, together with the protocol’s limitations.

A good score does not mean “usable”

The time-tracking example came from another campaign. Its documents could appear convincing while the actual startup failed. Reading explanations and running the product answer different questions.

We therefore strengthened checks that use the application’s actual entry point. For a booking, that means trying the journey from the screen and then finding the saved data, rather than checking only an isolated function.

Comparing the same copies before the same judge

A score can also change because the judge becomes stricter. We submitted two anonymised versions of the same subject to the same AI judge, with a randomised presentation order. The newer version was preferred on six of the eight subjects.

That result concerns a comparison between two versions of our method. It does not rank Maestro against every application-building tool or predict your project’s outcome.

What is public and what remains internal

We publish summaries: results, examples of faults and measurement limitations. The detailed reports, tested copies and full verdicts remain internal. Readers therefore do not have all the material needed to reproduce these campaigns here.

Our team produces these observations with AI judges. They are not an independent evaluation. Describing a defeat adds context; it does not automatically make our protocol impartial or exhaustive.

What to ask for on your project

Ask what was tried, what failed and what still needs checking. Clarify whether the result covers an initial set of features or the whole application. A useful report lets you locate a problem and replay it.

Our guide to checking an application without coding offers a starting point. Keep trial spending alongside the outcome too: it is meaningful only in relation to what was built.