Measuring AI visibility honestly

A tracker that asks ChatGPT once whether it recommends your brand and reports “yes” has measured one draw from a distribution. Named once in one run reads as 100%, and the honest interval around it is 21%–100%: almost anything. Named in 3 of 5 runs reads as 60% with an interval of 23%–88%. Only at 60 of 100 runs does the interval tighten to 50%–69%.

Three things follow. A rate is only meaningful with its interval and its n. The interval is set by how many runs you can afford, so sample size is a decision made before measuring. And a change between two weeks is a claim that needs a test, because two intervals that overlap are usually the same number.

We use the Wilson score interval. Unlike the textbook normal approximation it behaves at the edges (0 of 5 is not “0% ± 0”) and at small n, which is exactly where AI sampling lives. Its half-width shrinks with the square root of n: four times the runs halves the interval.

Five runs a week on one prompt give an interval of about ±33 points. That is a trend line over months, not a weekly reading. Pooling five weeks brings the same prompt to ±18 points.

Decide the precision first. runsNeeded inverts the Wilson half-width exactly, so it is the smallest n whose interval is at most the chosen width; it is slightly below the textbook formula z²p(1−p)/h² (93 runs against 97 for ±10 points at 50%).

±5 points on every prompt is expensive (381 runs each). The practical answer is to pool: prompts that share an intent on one engine measure the same thing and can be pooled; weeks can be pooled while the prompt set is stable. The sample-size calculator does the arithmetic for any precision, rate and prompt set.

A move from 51% to 63% on 35 runs a week looks like progress. A two-proportion z-test gives p = 0.334: not significant, and the product marks it n.s. The same difference held over eight weeks (144 of 280 against 176 of 280) gives p = 0.006: significant, with a 95% interval on the difference of 3% to 19%.

Counts need a different test. Citations, bot visits and AI referrals are events, not proportions, so the product uses an exact conditional test for two Poisson counts. Twelve citations one month and twenty the next is a ratio of 1.67 with a 95% interval of 0.83–3.36 and p = 0.215: not significant. The same ratio at 120 against 200 is significant (p < 0.001), interval 1.33–2.09.

A workspace with 35 prompts on 7 engines compares 245 rates every week. At α = 0.05, noise alone would flag about one in twenty of them. The product controls the false-discovery rate with the Benjamini–Hochberg procedure before any prompt is reported as moved.

Engines are reported separately, never averaged into one score. They differ in how they answer, cite and recognise brands, and the intervals make that visible: a brand can be steady on Perplexity and noise on Gemini in the same week.

AwesomeSEO samples the way this paper describes. The settings are fixed so weeks compare like with like.

A number that leaves the product carries seven fields. Anything less, and the reader fills the gaps with optimism.

Mention rate

Share of sampled runs in which the brand is named, for one engine and one prompt set.

Wilson interval

A confidence interval for a proportion that stays inside 0–1 and is accurate at small n (Wilson 1927).

Half-width

Half the distance between the interval’s ends; the “± points” a rate is reported with.

Two-proportion test

A z-test on the difference between two rates, with a Newcombe interval on the difference.

Count test

An exact conditional (binomial) test comparing two Poisson counts, with a ratio and its interval.

False-discovery rate

The expected share of flagged changes that are noise; Benjamini–Hochberg keeps it at q.

n.s.

Not significant at α = 0.05: the direction is shown, the change is not claimed.