Tackling human-written disinformation and machine hallucinations

It’s widely acknowledged that 2024 will be the biggest year for elections in human history, as nearly half the world’s population heads to the polls. This unique convergence of voting includes highly anticipated races in India, the UK, the European Union, and, of course, the made-for-television drama that airs every four years in the United States.

This year is also distinctive due to recent advancements in artificial intelligence that have stoked fears about machine-generated disinformation and deep fakes and their potential to mislead voters.

There is no lack of concern in the media. Many stories have considered how AI might be used by nefarious actors — or, perhaps, by even well-known and respected candidates — to gain advantage over opponents.

But is all lost? When we look ahead, is there any hope that the average voter will be able to navigate a hostile information environment in which shadowy villains with paid ChatGPT accounts have the power to determine the political future of nations and the world?

Professor Preslav Nakov

Preslav Nakov, department chair of natural processing and professor of natural language processing at MBZUAI, believes that though there are risks related to artificial intelligence and its ability to generate disinformation, our future might not be so bleak.

“I’m optimistic that this isn’t going to get out of control,” Nakov said. Recently, he has authored several studies that propose ways of identifying and combating the potentially harmful uses of large language models to generate false and misleading information.

Machine versus machine

In language, as in life, the only constant is change, and the way people describe false information has evolved over time.

Propaganda, which is a kind of communication designed to influence people with a mix of both true and false information, is certainly not new, with examples dating back to the ancient world. The term “fake news” came to prominence during the 2016 U.S. presidential election. But “fake news” doesn’t necessarily describe a key component of creating misleading information: “The problem isn’t just that fake news is false, but that it is weaponized,” Nakov said.

The phrase “disinformation” captures both elements of falsity and harm. “An important characteristic of disinformation is that the purpose is not just to lie, but also to persuade,” Nakov said.

Regardless of what it is called, disinformation can be written by humans or by machines. In each case, there are identifiable differences in the language based on whether it was created by a person or by an algorithm. For example, when people compose disinformation, they often employ specific rhetorical devices, and these techniques are commonly found in propaganda.

While researchers have made advancements to detect machine-generated disinformation with the help of artificial intelligence, AI programs struggle to identify it when it is written by a person. What’s more, differences in style and intent between human-written and machine-generated disinformation hamper the transferability of detection techniques from one kind to the other.

Recognizing the distinctive qualities of human-authored disinformation, Nakov and colleagues developed an innovative approach to detect it. They used a large language model to generate text that is styled according to the rhetorical techniques employed in real propaganda written by humans. By incorporating these linguistic tactics, such as appeal to authority and loaded language, the generated training examples align more closely with strategies employed by human propagandists.

This text is then used to train an application to detect human-written propaganda with the goal of improving the model’s the performance in this task. Nakov explained: “We generate fake news that makes intentional use of propaganda techniques, thus better mimicking human-written fake news. As a result, training on this machine-generated fake news yields an AI system that is better at detecting human-written fake news.”

The integration of a technique called self-critical sequence training, which allows a model to learn from the text it generates, is an important element of the team’s approach.

To evaluate the effectiveness of their framework, the researchers compiled a training dataset they call PropaNews that consists of more than two thousand examples. The scientists meticulously curated the data to encompass a wide range of human-written disinformation articles from a diverse array of sources.

Their intention is that PropaNews will serve as a valuable resource for future research and development in the field of disinformation detection.

Identifying hallucinations

Even if LLMs aren’t used to intentionally generate disinformation, they can still create text that is false. When large language models do this, they are said to “hallucinate,” a term that was named the 2023 word of the year by Cambridge Dictionary.

Nakov and colleagues recently created a program called Factcheck-GPT that can determine the factuality of LLM-generated text and identify machine hallucinations. Though there currently are AI-driven tools, such as FacTool and FactScore, that are designed to identify false claims by large language models, these applications struggle to do so.

“We want the information that a model provides to be truthful, unless we have asked it to generate something that is fictional,” Nakov said. “Until we solve the factuality issues with LLMs, in that they have the potential to make up information that is not true, we won’t be able to make full use of them,” Nakov said.

A whole new world

“We don’t know yet what percentage of content on the web is machine-generated, but we know that the amount of machine-generated content will grow over time,” Nakov said.

The expansion of machine-generated writing is not necessarily a bad thing, as large language models can create text that is truthful and informative for people. Yet, there are many potential misuses of large language models.

Nakov, along with other scientists from institutions that include Stanford University, Carnegie Mellon University and Meta, wrote a paper that outlines how we might address these challenges following the diffusion of large language models and their use by more and more people.

The study outlines technological innovations and educational and public policy approaches that can reduce threats posed by LLMs. Nakov noted: “Dealing with misinformation is an extremely complex problem and we need to add safety and fact-checking mechanisms into social media platforms and large language models.”

“But what also can have a major impact is raising awareness and educating people about the risks,” he said. “If we think that something that we read might be machine-generated, or that it might be fake news, simply being aware of that and considering the information with healthy skepticism can go a long way.”

As we approach these pivotal elections, it’s clear that AI will play a role in shaping public opinion. The battle against misinformation, whether human or machine-generated, is complex. Yet, with continued research and increased awareness, scientists like Nakov and his colleagues hope to build tools that will sharpen our perception of the world around us instead of obfuscating it.