Why we run every prompt ten times
The same question, asked twice, disagrees with itself. What that means for anyone selling — or buying — AI visibility.
Open ChatGPT. Ask it "best managed IT provider in Denver." Note the names. Open a fresh chat and ask the identical question. The list will be different — names appear, vanish, reorder.
This isn't an occasional glitch; it's the normal behaviour of these systems. The largest public test of it collected 2,961 responses across ChatGPT, Claude and Google's AI surfaces, running the same brand-recommendation prompts 60–100 times each. The same prompt produced the same brand list less than 1 run in 100. Identical ordering was closer to 1 in 1,000.
SparkToro / Gumshoe.ai (Rand Fishkin, Patrick O'Donnell), January 2026
What that breaks
It breaks the screenshot. If a vendor shows you a single chat where your brand appears (or doesn't), they've shown you one draw from a distribution that reshuffles on every pull. Run it again and the "proof" evaporates. An entire tier of the AEO industry currently sells this coin flip as an audit.
What survives
The same research contains the useful half of the finding: while the lists shuffle, each brand's frequency of appearing across many runs is fairly stable. The volatility lives in each individual answer, not in the underlying rate.
Frequency is a measurable, comparable, repeatable quantity. So that's what we measure:
- every buyer prompt runs repeatedly, on six engines (ChatGPT, Claude, Perplexity, Gemini, Google's AI Overviews and AI Mode)
- results are reported as frequencies with 95% confidence intervals — the honest error bars for the sample size
- every raw answer is stored verbatim, so any number can be audited down to the responses it came from
The confidence interval is the honesty
A brand mentioned in 3 of 10 runs isn't "30% visible" in any precise sense — at that sample size the honest range (a Wilson interval) is roughly 11–60%. We print that range. It's less satisfying than a single crisp number, and that's exactly the point: crisp numbers from small samples are how this industry lies politely.
When the interval narrows month over month and the frequency climbs, you're looking at real movement. When the intervals overlap, we say "no confident change yet" — in those words, in the report.
What to do with this
If you're evaluating any AI-visibility vendor — including us — ask one question: "how many times do you run each prompt?" If the answer is one, you now know what you'd be buying.
And if you want your own number, measured properly: the free six-page snapshot runs ~18 of your stakeholders' real questions, three times each, across four engines. A human reviews it before it sends.