Six days of testing: what we measured against BMAD

From August 5 to 10, 2026, we compared versions of our method with BMAD and then with each other. The results concern documents and an initial set of features, not complete applications delivered to customers.

A benchmark is an organised trial that compares results under defined conditions. Ours mainly revealed a difficulty: a document can score well while the software remains incomplete. Here are the main campaigns and what their figures allow us to conclude.

The scope: prepare, then build an initial set of features

The subjects were test projects: habit tracking, shared expenses, room bookings and inventory, for example. We prepared the documents, then developed an initial set of features, with the checks and repairs specified by the protocol.

The scoring rubrics also covered rules described in documents, including when the corresponding feature had not yet been built. A score out of 100 is therefore neither a test pass rate nor the proportion of a product ready for delivery.

August 6: nine subjects compared with BMAD

The version of our method evaluated that day averaged 89.5 against 83.5 for BMAD-METHOD v6 across nine subjects. It scored higher in eight comparisons; BMAD scored higher on room bookings.

The same rubrics and three AI judges per copy were used. The copies were anonymised. On the BMAD side, these were copies retained from previous campaigns and assessed by additional judges, not nine executions rerun at the same time as ours.

Recorded execution costs were $115.65 for our method and $238.66 for the corresponding BMAD executions. Our process reached its endpoint in nine cases, BMAD’s in six. This describes completion of the trial process within its spending limits: features and faults still remained to be addressed.

August 10: eight subjects in another campaign

A later version scored 93.3 across eight subjects, against a BMAD reference of 85.6 for that group. The 93.3 figure includes a new assessment of the shared-expenses project, with context for a change requested during the trial. Using only the initial need, the average was 92.5.

The eight executions totalled $105.64. The spending for this campaign and its exclusions are explained separately. The 93.3 score must not be combined with the nine subjects and costs from the August 6 campaign.

Why we changed how we compare results

A more recent version subsequently received an average of 87.0. To understand the difference, we asked the same AI judge to compare two anonymised copies of the same subject in a randomised order. The recent version was preferred on six of the eight subjects.

Those comparisons revealed faults hidden by earlier high scores, including a time-tracking application that would not compile. We strengthened execution checks. Scores from different campaigns alone are not enough to measure progress.

What to take from this

This is a summary of internal reports written by the Maestro team, using AI judges. The complete copies and verdicts are not published here. There are few subjects, generated results vary, and the protocol is not an independent audit.

These results do not rank other AI tools or guarantee your application’s price, security or quality. For your project, ask above all for a check of the actual usable journey and a list of unfinished work. We also describe the failures that changed our checks.