Understanding AI-generated misinformation and evaluating algorithmic and human solutions

by Machine Learning Center at Georgia Tech

21 April 2023

share this:

Jiawei Zhou, a PhD student in Georgia Tech’s School of Interactive Computing.

Existing machine learning (ML) models used to detect online misinformation are less effective when matched against content created by ChatGPT or other large language models (LLMs), according to new research from Georgia Tech.

Current ML models designed for, and trained on, human-written content have significant performance discrepancies in detecting paired human-generated misinformation and misinformation generated by artificial intelligence (AI) systems, said Jiawei Zhou, a PhD student in Georgia Tech’s School of Interactive Computing.

Zhou’s paper detailing the findings has received a best paper honorable mention award at the 2023 ACM CHI Conference on Human Factors in Computing Systems. Advised by Associate Professor Munmun De Choudhury, Zhou’s research demonstrates that LLMs can manipulate tone and linguistics to allow AI-generated misinformation to slip through the cracks.

“We found the AI-generated misinformation carried more emotions and cognitive processing expressions than its human-created counterparts,” Zhou said. “It also tended to enhance details, communicate uncertainties, draw conclusions, and simulate personal tones.

“We’re one of the very first to look at this risk. As more people started to use ChatGPT, they’ve noticed this problem, but we were one of the first to provide evidence of this risk. And there are more efforts needed to raise public awareness about this potential and call for more research efforts to combat this risk.”

Zhou started exploring GPT-3 in 2022 because she wanted to know how one of the early predecessors to ChatGPT would handle prompts that included misinformation about the Covid-19 pandemic. She asked GPT-3 to explain how the Covid-19 vaccines could cause cancer.

“The results were very concerning because it is so persuasive,” Zhou said. “I had been studying informatics and misinformation for a time, and it was still persuasive, even to me. The output would say, ‘It can cause cancer because there is this researcher at this institute, and their research is based on medical records and diverse demographics. The research supports this possibility.’ The writing of it is so scientific.”

Zhou and her collaborators accumulated a dataset of human-created misinformation, including more than 6,700 news reports and 5,600 social media posts. From that set, Zhou and her team extracted the most representative topics and documents of human-generated misinformation. They used those to create narrative prompts, which they fed to GPT and recorded the output.

Both the GPT-generated output and the original human-created dataset were used to test an existing misinformation detection model called COVID-Twitter-BERT (CT-BERT).

Jiawei Zhou and Munmun De Choudhury.

Zhou said while the human- and AI-generated datasets were intentionally paired, a statistical test showed there are significant differences in detection model performance.

CT-BERT experienced a decline in performance in detecting AI-generated misinformation. Out of 500 prompts based on AI-generated misinformation, it failed to recognize 27 as false or misleading, compared to missing only two from the human-generated prompts.

“The core reason is they are linguistically different,” Zhou said. “Our error analysis reveals that AI misinformation tends to be more complex, and it tends to mix factual statements. It uses one fact to explain another, though the two things might not be related. The tone and sentiment are also different. And there are less keywords that detection tools normally look for.”

Zhou’s experiments showed that GPT could use information to create a news story using objective, straightforward language and use that same information to create a sympathetic social media post. That points to its capability of changing tone and tailoring messages.

“If someone wants to promote propaganda, they can use it to customize a narrative toward a specific community,” Zhou said. “That makes the risk even greater. It shows that it has some flexibility to alter its tone for different purposes. For news, it can sound logical and reliable. For social media, it conveys information quickly and clearly.”

As LLMs continue to rapidly grow and expand, so do the risks of misinformation.

ChatGPT operates on Open AI’s GPT-3.5 and GPT-4 models, the latter of which was released on March 14. Since ChatGPT was released, Zhou has given it the same prompts she gave to GPT-3. The results have improved with some corrections, but the latter has the advantage of having more available information about Covid-19, she said.

Zhou said steps should be taken immediately to evaluate how misinformation detection tools can adapt to ever-improving LLMs. She described the situation as an “AI arms race,” and the tools that are currently used to combat misinformation are well behind.

“They are improving the generative capabilities of LLMs,” she said. “They’re more human-like, more fluent, and less and less distinguishable from human creations. We need to think about ways we can distinguish them and how we can improve our misinformation detection abilities to catch up.”