Does an AI idea validator give the same verdict twice? We tested ours on two versions

Six texts, all written by us as examples, never from a real founder. Three were plain ideas: a night-shift nurse's sleep plan for rotating rotas, a vet clinic's take-home instructions for pet owners, and a compliance tracker for small fishing fleets that a free official app already partly covers, with certificate reminders. Three were the nurse and vet ideas again, each with a test result added at the end. Each text ran 4 to 8 times.

As of 5 November 2026, Idea Validator and Roast My Idea reach their verdict through the same grading engine. We ran that engine from a copy of the current code, with the same settings and its search for rivals on. We skipped the question round, the few questions the tool can ask before a run when the text leaves something out. We used no project history, and every text was in English.

• The nurse idea came back Sharpen it in 7 of 7 runs, the vet clinic idea in 8 of 8, and the fishing fleet idea in 6 of 6.
• Every one of those 21 reads asked for the same kind of next test: name a price, show it to between 1 and 10 named people, and count who says yes or pays by a fixed date four weeks out. On the nurse and vet ideas the people and the counts changed from run to run. On the fleet idea it was always the one fleet owner the founder knows. The kind of test did not change.
• Each read also names the one thing most likely to kill the idea. On the fleet idea it was the free official app in 6 of 6 runs, and on the nurse idea it was an existing sleep app for shift workers in 7 of 7. On the vet clinic idea it was always an existing tool, but two kinds of tool: the clinic software vets already use in 3 runs, AI note tools in

Evidence did. We added a test result strong enough for the rules' Build line to the vet clinic idea. In our texts that meant strangers who paid, a group the founder could list, a version the founder could deliver, and no false claim that nothing else does this: 6 of 20 clinics the founder had never worked with agreed to a paid pilot, and 2 paid. Build it, 5 of 5. We did the same with the nurse idea: 31 nurses from other hospitals sent their next rota, 9 paid up front and 6 came back without being asked. Build it, 5 of 5. After a Build it, the next test changed too: compare the product with the tool the customers use now or used before, and count who says what it does better.

A failed test did not turn the nurse idea into Kill it. With 12 free plans handed out, nobody paying and nobody coming back, the read stayed Sharpen it in 4 of 4 runs and asked those nurses for money again, at a count and a date. That is your riskiest assumption put back in front of you.

On 20 October we planned 112 runs of an earlier version on nine base ideas. 107 completed, the other 5 did not finish, and none of the nine ideas got two verdicts. Ten of those runs chose the flaw from a fixed list, a setup the product does not use, and on the vet clinic idea 8 of that day's 15 Validator reads were not clear enough to show. The fishing fleet idea above and a home kitchen in a place where the law mostly does not allow selling home-cooked meals came back Kill it in all thirty of their runs. We also ran texts with evidence added. Evidence that strong gave Build it every time, but the two texts with a weak or failed test split 4 to 1: the failed nurse test came back Kill it 4 times and Sharpen it once, and a weak vet test came back Sharpen it 4 times and Kill it once. We did not rerun that weak vet test on the current version.

The current version keeps Kill it for a short list of cases its rules name. The founder says the business deceives or knowingly breaks a law or a platform's rules. A law forbids it, or the permit cannot be had. The founder will not put it in front of real customers. A real customer test found no problem and a priced offer was refused. Or the read finds three things at once, each one stated, not just missing: a pain nobody is named as having, no one shown a price, and a claim that nothing else does this which a rival disproves.

None of those fits our two texts. The fleet idea has a real if small pain, so it now comes back Sharpen it, with one payment test on the one fleet owner the founder knows. That owner has already said he would not pay for a second app, so the test is cheap and a no is likely. In the failed nurse test the pain was real, so no Kill rule fired, and the read asks for that price again.

• On these six texts, a second run on the current version never changed the verdict. That is a small test, so treat it as a pattern, not a promise. A rewrite that sounds better is not evidence.
• Read the named flaw as one candidate. If the idea has two rivals, two runs may name two of them, and both are real.
• Treat the test it asks for as the real output. A paid pilot, a stranger who pays, a person who comes back unasked: those moved the verdict on both versions.

Six texts on one day, 4 to 8 runs each, English only, and our own engine. The ideas and the evidence were written by us, the evidence to meet or miss a bar, so it was cleaner than a real result, and a real founder's rougher text may behave differently. We skipped the question round, which can change what the engine reads. We did not test other products, and we did not test whether a different wording of the same idea gets a different verdict. The numbers above describe these two tests, not AI idea tools in general.

• nommi internal test, 5 November 2026: 35 runs of the current grading engine on six example texts, with the run files kept internally.
• nommi internal test, 20 October 2026: 112 planned runs, 107 completed, on nine example base ideas on an earlier version of Roast My Idea and Idea Validator, plus separate runs of the same ideas with evidence added, with the run files kept internally. Its counts were recounted from the run files on 5 November 2026.