If you ever thought that Google’s AI Overviews feature was wrong enough as is, well, a study proves that it may be more than what you think.
A recent analysis by AI startup Oumi for The New York Times (NYT) found that Google’s AI Overviews (essentially the overviews that appear above Google Search results) were correct about 9 out of 10 times.
That seems pretty impressive until you realize that Google processes roughly five trillion search queries per year, and the analysis found that AI Overviews provides tens of millions of incorrect answers every hour and hundreds of thousands every minute.
The New York Times noted that more than half of the accurate responses were “ungrounded,” meaning they linked to websites that didn’t fully support the information they provided. The analysis noted that this makes it difficult to verify the accuracy of the AI Overviews. This, as noted by both Futurism and The New York Times, has created a bit of a misinformation crisis.
To conduct the study, Oumi analyzed the accuracy of AI Overviews using a benchmark test called Simple QA, which is widely used to measure an AI system’s accuracy. Oumi used Simple QA on both Gemini 2 in October, and Gemini 3 in February, once it was upgraded. In both cases, the analysis focused on 4,326 searches, with the Gemini 2 being correct 85 per cent of the time and the Gemini 3 95 per cent of the time.
With all this in mind, Google acknowledged the issue but has downplayed its importance. “Our Search AI features are built on the same ranking and safety protections that block the overwhelming majority of spam from appearing in our results,” said Google Spokesperson Ned Adriance to the NYT. “It doesn’t reflect what people are actually searching on Google.”
Image credit: Shutterstock
Source: The New York Times
MobileSyrup may earn a commission from purchases made via our links, which helps fund the journalism we provide free on our website. These links do not influence our editorial content. Support us here.
