AI search engine news query error rate exceeds 60%, and reliability is worrying

A new study reveals serious problems with AI search engines when processing news information. The Digital News Center of Columbia News Review (CJR) tested eight AI tools with real-time search capabilities, and the results were worrying: More than 60% of news queries received incorrect answers .

The study found that the error rates of different AI platforms differ significantly, with Grok3's error rate as high as 94%, and ChatGPT Search also reached 67%. Even paid versions, such as Perplexity Pro and Grok3's premium services, give more frequent confident but incorrect answers. The researchers stressed that these AI models have a widespread tendency to be “confidently wrong”, that is, when there is a lack of reliable information, they do not refuse answers, but instead fabricate seemingly reasonable wrong answers.

In addition, the research also reveals the citation problems of AI search engines. They often point to joint publishing platforms for news content, rather than the original publisher’s website . What is even more worrying is that some AI tools will also fabricate invalid URL links , causing users to be unable to access the information source. For example, more than half of Grok3's reference links are invalid in the test.

These issues present serious challenges for news publishers. Blocking AI crawlers may cause the content to lose its signature completely, while allowing crawling may face the dilemma of the content being widely used but not being able to be diverted back to its own website. Times chief operating officer Mark Howard expressed concerns about transparency and control.

The researchers noted that given that about a quarter of Americans have now used AI models as a replacement for traditional search engines, such a high error rate is worrying about the reliability of information. The study further confirms a similar report released last November, which also points to the accuracy issues that ChatGPT has when dealing with news content. Although OpenAI and Microsoft acknowledged that they received the research results, they did not directly respond to specific questions.