We asked the same 20 questions twice. All 20 gave a different answer

Last week we ran an audit for a European lab-services company: twenty questions a buyer would ask before choosing a supplier, put to an AI assistant with web search, to see whether the company appeared in the answers. To check the method, we asked every question twice, about a minute apart, on the same day, with nothing changed.

All twenty pairs came back different. Not slightly reordered, different: every pair named a different set of companies. In thirteen of the twenty, the company named first changed. The median pair differed by four or five companies, and the overlap between the two lists was 0.40, meaning less than half the names were common to both runs. The pages the model retrieved differed in all twenty pairs too, by a median of seven domains.

Those are our own numbers, from our own audit, computed from the raw runs. The company is not named here.

Why the same question gives different answers

The instinct is to assume the model is being inconsistent. That is not really what happens.

When you ask an AI assistant a question it cannot answer from memory, it does not answer directly. It writes one or more search queries, sends them to a search engine, gets back a list of results, opens some of them and writes an answer from what it read. The wording of the queries it invents can vary between runs. The search results themselves change from minute to minute. Which of those results the model chooses to open varies. And only then does the model's own sampling add variation on top.

So the answer is the last step in a chain where almost every earlier step is unstable. In our runs the retrieved domains differed in every single pair, which is the mechanism: two different reading lists produce two different answers. The model was not being moody, it was reading different pages.

This is also why the variation is not evenly spread. A question with a clear, well-indexed answer tends to be stable. A question like "which suppliers should I consider in Europe", where dozens of companies could reasonably appear and no page is authoritative, is exactly the kind where the retrieval step has room to wander, and those were the questions that moved most in our data.

What this means if you are checking your own visibility

Most people check the way you would expect: they open ChatGPT, ask whether their company comes up, get an answer, and form a view. Our data says that view is close to worthless on its own.

If a single run has a roughly even chance of naming you, then asking once tells you almost nothing. Ask on a good run and conclude you are fine. Ask on a bad run and conclude you are invisible. Both conclusions come from the same underlying reality, and both are wrong, because the real answer is a probability, not a yes or no.

The same applies to the reverse mistake. A company that appears in one screenshot has not "won" the answer, and a competitor who appears once is not dominating it. Without repetition there is no way to tell a stable position from a lucky one.

What we do about it

Three things, and none of them are clever, they are just the discipline the variance demands.

Repeat every question. A single run is an anecdote. We ask each question multiple times and treat the result as a rate, not a fact: named in six of ten runs, not "named".

Report the spread, not one number. A visibility figure with no sample size behind it is a screenshot with a percentage on it. The useful output is how often you appear, out of how many runs, on which day, with the questions written down so the check can be repeated.

Look at what was retrieved, not only what was said. Because the variation comes from the retrieval step, the reading list is more informative than the answer. If your page is never among the pages opened, no amount of rerunning will change the outcome, and that is a fixable problem. If your page is opened but you are still not named, that is a different problem with a different fix.

The honest limits of this

This is twenty questions, one company, one model with web search, one day, from one country. It is enough to show that identical runs differ, which was the point, and it is not enough to say how much they differ in general, or whether other assistants vary more or less. We only ran repeat pairs on one of the three assistants in that audit.

There is a related published finding worth knowing: when Orbit Media tracked 13,184 citations across four AI tools, the four agreed on citing the same domain for the same question only 1.7% of the time. (Orbit Media, LLM Citation Study) Different tools read almost entirely different pages. Our result says the same tool does too, between one minute and the next.

The question on the card stands: if the list moves every time, what does a ranking mean? It means a rate, measured over repeated runs, or it means nothing.