Study warns that repeatedly pressuring AI chatbots can increase misinformation

12:45

By: Azzat Manal

Study warns that repeatedly pressuring AI chatbots can increase misinformation

A new study has raised concerns about the reliability of artificial intelligence chatbots, finding that repeatedly challenging or pressuring them to accept false information can sometimes increase the likelihood that they will eventually agree with misleading claims.

Researchers from the University of Arizona tested seven AI models, including GPT-3.5, GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 70B and DeepSeek-R1. Rather than evaluating a single response, the researchers conducted extended conversations in which the models were repeatedly exposed to incorrect statements.

The experiment involved 100 false claims covering a broad range of subjects, from obviously nonsensical assertions to more ambiguous statements. Researchers repeated the misinformation as many as 50 times within the same conversation to determine whether the systems would eventually change their responses.

None of the seven models proved completely resistant to the effect. GPT-3.5 showed the highest rate of agreement with false claims, accepting them in 12.3% of interactions. Claude 3.5 Sonnet recorded the lowest rate, at just 0.08%, while GPT-4o and GPT-4o-mini remained below 1%.

The researchers found that GPT-3.5 initially rejected 96 of the 100 false claims tested. After the claims were repeated 50 times, however, the model eventually agreed with 18 of them.

Another pattern identified in the research was what the researchers described as “vacillation,” in which some AI systems repeatedly switched between accepting and rejecting the same misleading information.

The study also suggested that ambiguity can make AI models more vulnerable to misinformation. Claims concerning topics with limited or unclear information available online were more likely to be accepted during repeated interactions, with the researchers identifying a statistically significant relationship between ambiguity and susceptibility to misleading claims.

The researchers further examined whether more confrontational exchanges could influence the models. Most systems remained relatively stable, but DeepSeek-R1 showed a marked increase in its acceptance of misinformation when subjected to argumentative pressure. Its acceptance rate rose from 1% during simple repetition to 22.2% under more confrontational conditions.

The use of sarcasm and repeated humorous responses also made some interactions more difficult for researchers to classify consistently.

There was, however, an encouraging finding when the models were given an opportunity to reconsider previous mistakes. GPT-4o, GPT-4o-mini, Gemini 1.5 Pro and DeepSeek corrected all of the errors they had previously made. GPT-3.5 corrected only 32% of its earlier mistakes.

Claude 3.5 Sonnet made very few errors during the initial tests, but it failed to correct the four mistakes it did make.

The researchers nevertheless cautioned that the study’s relatively small sample size limits the strength of the conclusions that can be drawn. The findings nonetheless highlight a broader challenge for generative AI: chatbots can sometimes prioritize conversational consistency or responsiveness over maintaining a firm rejection of inaccurate information.

As AI systems become increasingly integrated into education, research, work and everyday decision-making, the study underscores the importance of verifying important information rather than assuming that a confident or repeated chatbot response is necessarily accurate.