What is measured.
Named AI assistants, under stated test conditions. The businesses and places in our questions are what we ask about. The assistants are what we score.
Method
How we measure.
Two kinds of study.
Accuracy studies check what an answer claims. Each claim is marked correct, incorrect or unverifiable. Source studies record which sources an answer used. A source is cited when the answer attaches it to a specific statement, and retrieved when the assistant fetched it but did not attach it to anything. Where an assistant does not show what it fetched, we record that as not observable and do not guess.
Written down first.
The questions, the reference facts and the scoring rules are recorded before the first question is asked, and are not changed afterwards. Any later change is added as a dated amendment, never by editing what came before.
Three verdicts.
Each claim in an answer is marked correct, incorrect or unverifiable. A claim we cannot check against a reference fact is unverifiable, not wrong. An invented detail, such as a price where none is published, is reported separately from an error.
What every answer records.
The assistant, the exact model, whether search was on, the effort setting where the assistant has one, the date, the language and whether memory and personalization were on or off. We test with memory and personalization off unless a finding says otherwise.
Questions in other languages.
Questions in languages other than English are written the way a local person would ask, not translated word for word. Every finding says who wrote them and how they were checked, including when the writing or checking was done by an AI model, and which model.
What does not count.
A call that failed, timed out or was refused is not an observation about the assistant. It is excluded from every rate and the number of failures is reported alongside.
Answers are kept.
Every answer collected is archived and never overwritten.
Limits.
We test through each company's developer access, which may behave differently from the consumer app. A finding describes the tests we ran on the dates we ran them. It does not predict every future answer.
Found something we got wrong? [email protected]