A free AI assistant delivering code that did not compile: three tuning passes

We wanted a route without a subscription. We put a free assistant on our test bench, and it delivered twenty-five approved stages whose code would not launch. What three tuning passes taught us, about it and especially ourselves.

A free AI assistant can approve every methodological stage and deliver code that does not compile: our August 17, 2026 test bench measured it (internal team notes). Mistral Vibe handled 1.44 million tokens without a cutoff, had 25 stages approved and delivered a project that would not start.

Why we built this bench

A paid assistant plan can be an initial expense for directing agents in Maestro, which is free during beta. This test concerns that AI expense, not a product's full cost: your approval time, its hosting, external services and maintenance must still be planned for. We wanted a way to begin without a paid plan for a first encounter by verifying an existing free tier without a card. Our bench runs the engine for real on disposable projects, with an agent playing the user as in our five tester temperaments. We only needed to switch the working assistant and see where the chain broke.

First pass: eighteen stages instead of three

The first express-mode trial produced correct brief, requirements and architecture, all approved by the simulated user. Then breakdown returned 18 stages where express expected three, exceeded its deadline and stopped before any code. No rate blockage, however: free was not throttled, just slow, each stage taking two to five times longer.

We read the transcript. The explosion started in specifications, five major work packages instead of two, and express merely said ‘focus on essentials’. For one assistant this is a constraint; for another, a style. The restraint we thought established was spontaneous behaviour from one assistant, written nowhere. We wrote the rule: essentials mean a budget, at most two major packages and five stages, reduce scope. The rerun finished in 19 minutes with two packages and five approved stages.

Second pass: the entire project would not launch

On a longer project, the same assistant lasted 93 minutes and 1.44 million tokens without cutoff, with 25 approved stages and contracts. The delivered code did not start: eight compilation errors, a wrongly specified entry point, no application assembler, a comment promising undone work and a nonexistent launch command. Yet every stage passed verification because verification checked stages separately, never the whole project. A second implicit rule appeared: an assistant familiar with our habits assembled everything independently, and we mistook habit for a guarantee. The design from this engine was clean and generic, with a bare form and externally loaded fonts.

Third pass, not yet rerun

We added another verification requirement: the whole project compiles and starts, its start command is executed then immediately stopped, and the manifest promises only existing things. Dry-run checks pass. The real-condition rerun has not happened: we write that instead of declaring resolution. This discovery benefited every assistant, including paid ones, as often happens when a method corrects what a better model masked.

What free is worth

The explanation for ‘zero cutoffs’ arrived from the provider by email that evening: the free tier is not rate-throttled but monthly capped. Our afternoon, around 1.9 million tokens, burned 80% of quota, and the key stopped at 100% until renewal. A free tier therefore serves trials, not work. We will never call it ‘use for free’ and will not make it the first encounter until the complete chain finishes twice consecutively.

In practice, to test without paying

Take the free key, make one short project and check two things: does breakdown produce a readable number of stages, and does the product launch on your machine at the end? These checks cost an evening and tell you everything. If the second fails, nothing else counts, however beautiful the documents: it is the prototype/product boundary, invisible on screen.