More than 60% of AI chatbots answer incorrectly, news credibility is challenged

The latest research from the Columbia News Review Digital News Center found that popular AI search tools provide incorrect or misleading information when answering questions with more than 60%. The researchers tested eight generative AI chatbots, including ChatGPT, Perplexity, Gemini and Grok, asking them to identify excerpts from 200 latest news articles. The results show that more than 60% of the answers are wrong, and these chatbots often fabricate titles, do not quote articles, or quote unauthorized content.

Disappointingly, these chatbots rarely express uncertainty, but instead provide wrong answers with inappropriate confidence. For example, ChatGPT provides 134 error messages in 200 queries, but has expressed suspicion only in 15 times. Even the paid versions of Perplexity Pro and Grok3 are not satisfactory, with the number of wrong answers higher, although they are priced at $20 and $40 per month, respectively.

In terms of content citations, multiple chatbots failed to follow publisher restrictions, and five chatbots even ignored the widely accepted standard of bot exclusion protocols. Perplexity once correctly quoted articles from National Geographic when publishers restricted their crawlers. Meanwhile, ChatGPT recited USA Today articles of paywall content through unauthorized Yahoo News.

In addition, many chatbots direct users to reprinted articles on platforms such as AOL or Yahoo, rather than the original source, even if an authorization agreement has been reached with the AI ​​company. For example, Perplexity Pro cited a reprinted version of the Texas Forum but failed to give the due signature. Grok3 and Gemini often invent URLs, and 154 of Grok3's 200 answers link to the error page.

This study highlights the growing crisis facing news organizations. More and more Americans are using AI tools as their source of information, but unlike Google, chatbots do not direct traffic to websites, but instead summarize content without linking back, causing publishers to lose advertising revenue. Danielle Coffey of the News Media Alliance warned that without control over the crawlers, publishers would not be able to effectively “make valuable content or pay journalists’ salaries.”