What our beta testers broke, second pass: the silence that blocked everything

After a first real user test, 21 written comments revealed a defect invisible internally: an entire step type was handled nowhere in the engine. The chain froze silently, waiting for an answer that never came.

After our first week of beta, a second real user test produced 21 written comments. The most serious issue was invisible in our own trials: an entire step type, the one that asks a question and waits for the answer, was handled nowhere in the engine. The chain stopped silently and never resumed.

The symptom: a chain freezing without saying so

Scoping a project sometimes asks the user a question before continuing. In one specific case, the brief began writing before the person had even read the question: the engine advanced as though the answer had already been given when it had not. A rushed tester might have noticed nothing until later discovering a document that did not match what they had actually meant.

Why it had never happened to us

Internally, a rarely triggered step type can easily slip through a code review focused on frequent cases. It took a real user in a real conversation, taking time to genuinely answer a real question, for the defect to appear, exactly the kind of scenario a hastily written internal test would have skipped without noticing. The fix imposes a simple rule: the chain stops and resumes without replaying a completed step, and when in doubt it advances rather than blocking an entire sequence indefinitely.

A choice removed rather than added

The same test revealed the opposite signal: an AI model selector, shown early in the journey, helped nobody decide. The beta feedback was clear: the engine's default is enough; most people neither want nor are able to judge between two AI models at that stage. The selector was removed from that precise point rather than improved: sometimes the best fix removes a choice instead of adding one. This goes against the usual product instinct, adding another option to satisfy everyone, and takes real discipline to maintain once calm returns.

What telemetry showed that emails did not

Written feedback says what someone felt; anonymous telemetry says what actually happened, step by step, before the blockage. Combining the two located the defect in one day instead of guessing from an approximate description. Without telemetry, the only clue would have been a frustrated user saying ‘nothing happens’, a phrase saying nothing about the exact point where the engine fell silent.

What remains open

Not every defect reported in a user test is corrected the same day. Some await a broader product decision, and we prefer saying so to pretending everything is resolved. It is the same principle as our benchmarks publishing our defeats: showing what broke matters as much as showing what works.