When the score falls: what our software comparisons showed

In August 2026, a newer version received a lower average score. We compared the software outputs in pairs to understand the result. Here is what that check showed, without converting the verdicts into a new score.

A score falling from 93.3 to 87.0 seems to suggest lower quality. But our two campaigns in August 2026 were not assessed under the same conditions. To understand the difference, we put the outputs from both versions side by side on the same scenarios.

Why the drop made us question the result

The 93.3 average concerned our method v2.9, including one case reassessed with a simulated client’s decision taken into account. The 87.0 average concerned v3.1, assessed in a different campaign. As our review of the tests explains, subtracting those averages could not directly measure an improvement or regression.

Comparing two outputs before the same judge

We organised eight paired comparisons: for each scenario, one AI judge examined both anonymised outputs, presented in random order. Version 3.1 was preferred in six cases out of eight. Version 2.9 retained the lead for invoicing and habit tracking. This result describes those eight comparisons, not a general success rate.

Defects missing from the initial score

Execution tests revealed problems in older outputs that had received high scores. A time-tracking application failed to compile; its startup code also reset the database. A file-organising tool could overwrite files. These findings showed that the initial scores had missed significant defects. They do not establish that jury strictness alone explained the entire difference between the averages.

Why we are removing the converted score

We had tried to translate the judge’s preferences into points using a scale fixed before the verdicts. The result depended on that scale and on the old scores chosen as reference points. It was a constructed estimate, not a new directly observed score. Here we retain the six favourable comparisons out of eight, without assigning them a score out of 100 or an equivalent numerical gain.

The remaining limitations

The protocol was internal: one retained run per version and scenario, one AI judge per comparison and only the first development batch. The eight cases demonstrate neither consistent results nor completed applications. The jury for the v3.1 campaign received the initial requirement and the one approved before development; in these eight cases, the two documents were identical.

A useful rule for your project

Before comparing two scores, check the requirement being assessed, the software version, the criteria and the tests actually run. If conditions change, comparing the software outputs directly can provide different information from the average. We retain unfavourable results as part of our approach to publishing defeats. A score helps you examine the work; it does not replace checking the features you need.