How to measure AI visibility without fooling yourself

The most common way teams check their AI visibility is also the worst: type a question into ChatGPT, screenshot the answer, and draw a conclusion. Run the same question five more times and you'll often get a different list. Measuring a non-deterministic system takes the same discipline as polling - and skipping that discipline produces confident, wrong conclusions.

Sample, don't spot-check

A single answer is an anecdote. Ask the same question many times and the noise averages into a signal: appearing in 8 of 10 runs versus 1 of 10 is a real, stable difference even though any individual run varies. Every credible number downstream - trends, competitor gaps, the effect of your changes - depends on this.

There's no universal "correct" sample size - it trades off against API cost and how fast an answer is to run. What matters is picking a number and holding it constant, because a mention rate computed from 5 runs one month and 50 the next isn't comparable, even if the method is otherwise identical.

Hold the method constant

• Label the provider and model. "AI visibility" isn't one number - GPT-5 with web search, Gemini, and Perplexity behave differently and should be reported separately.
• Keep prompts unbiased. Asking "is Acme good?" tells the model the answer you want. Ask what a buyer would ask, with no brand in the prompt.
• Version the methodology. When you change prompts, models, or sample size, mark the break - otherwise you'll read a method change as a market change.
• Timestamp everything. Answers drift when models update; a number without a date is unusable.

Measure more than mentions

Whether you're named is the headline, but two other dimensions decide what a mention is worth. Position: first-named products get chosen; fifth-named rarely do. Citations: which sources the answer drew on - because that list is where your visibility actually comes from, and it's the only part you can directly act on.

Track competitors in the same runs

Your mention rate alone has no scale. Appearing in 40% of answers is excellent if your nearest competitor appears in 10%, and alarming if they appear in 90%. Because competitors get measured in the very same sampled answers, the comparison is apples to apples by construction - you're not comparing your Tuesday sample against their sample from a different week with a different model version behind it.

A minimal setup you can build yourself

You don't need a platform to start - a script and a spreadsheet gets you a real baseline. The shape of it is a nested loop: for each prompt, for each provider, run N times and record the result.

for prompt in buyer_prompts:
  for provider in [openai, anthropic, perplexity, gemini]:
    for run in range(N):
      answer = provider.ask(prompt)          # no brand name in the prompt
      log(
        prompt, provider, run,
        mentioned = brand in answer.named_brands,
        position  = answer.position_of(brand),
        sources   = answer.cited_urls,
        timestamp = now(),
      )

That log is the whole dataset. Mention rate, position, and cited-source frequency are all just aggregations over those rows - the discipline is entirely in running it the same way every time, not in the analysis.

What a good report looks like

Provider-labelled mention rates with sample sizes, position when mentioned, the cited sources ranked by frequency, competitor rates from the same runs, and a methodology version plus timestamp on all of it. This is the shape we built Arcanoris's reports around - but the principles hold whether you use a tool or a spreadsheet and an API key.