Facts and fabrications: New insights to improve fake news detection

Although the phrase “fake news” rose to prominence towards the end of the last decade following the 2016 U.S. presidential election, people have always had to develop methods to separate fact from fiction. However, the recent rise of large-language models, with their ability to quickly generate huge amounts of both factual and fictional text, makes identifying fake news both more complex and more necessary today.

A recent study by Preslav Nakov, department chair of natural language processing and professor of natural language processing at Mohamed Bin Zayed University of Artificial Intelligence, and coauthors from Cornell University, considers how scientists can develop systems to detect fake news in a complex and evolving media landscape in which text, both true and false, can be generated by either humans or machines.

The study is the first comprehensive look at fake news detectors’ ability to identify both human- and machine-written real and fake news across a wide variety of scenarios. The research was recently presented at the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, which was held in Mexico City.

“The essence of the paper relates to how we should train fake news detectors when the proportion of both real and fake news written by machines will increase over time and how we can account for these changes when we train these systems,” Nakov said.

Shifting sands

In the recent past, most news, whether real or fake, was written by humans. There were some selected reports — like weather forecasts or summaries of sporting events — that were sometimes written by machines, Nakov explained. But machine-generated text was limited and specialized. And while prior to 2018 most fake news was written by humans, Nakov and his coauthors write, today there is a huge amount of misinformation found online that has been generated by large-language models, or LLMs as they’re called.

While LLMs can be used to generate text, they are also a powerful technology for analyzing it. “We leverage the same power that is used to generate text to detect machine-generated text,” Nakov said.

In previous research, Nakov studied LLM-based systems that analyzed the relationship between fake news and machine-generated text. The detectors he looked at had two biases, however. First, they had the tendency to classify machine-written news as fake, regardless of veracity. Second, they had a tendency to categorize human-written news, regardless of veracity, as true.

These tendencies limit the utility of these systems in an era where some real news is written by machines and some fake news is written by people. And in the years ahead, these biases would prove to be even more of a problem, especially with respect to real news. “In the future we can imagine a kind of machine dominance where the vast majority of real news will be written by machines,” Nakov said.

Even today, news agencies use LLMs for “legitimate purposes, such as assisting journalists in content creation,” Nakov and his coauthors write. In short, determining if a piece of text was written by a human or by a machine doesn’t necessarily provide information about its truthfulness.

To provide more specificity related to different types of text the detectors must analyze, Nakov and his coauthors set up a framework in their study comprised of four categories: machine-paraphrased real news, machine-generated fake news, human-written real news, and human-written fake news. Fake-news detectors must be able to process these different kinds of text and determine their authenticity based not on how text was generated, but rather on the information contained within it.

The study led to three key insights. First, if a detector is trained on human-written real and fake news, it will have the ability to detect machine-generated fake news. But if a detector is trained only on machine-generated fake news, it won’t be so good at detecting human-written fake news. Second, the researchers found that a detector would have “balanced performance” across the different kinds of text if the training data set had less machine-generated news when compared to the test set. Third, fake news detectors are better at identifying machine-generated fake news than they are at identifying human-generated fake news. The authors attribute this discrepancy to biases that arise from current detector training practices.

Nakov and his coauthors also found that larger models don’t always perform better than smaller ones. On certain subclasses of text, smaller models were more effective, perhaps because they weren’t biased in training in the same way as the larger models were.

That said, because many LLMs are so called “black boxes,” meaning their inner workings aren’t interpretable by people, researchers aren’t certain why they see the results they do. “We know what works well to train detectors based on their performance, but what they are doing to classify text is not very explainable,” Nakov said.

The games people play

Even with improvements to fake news detectors that are made possible by insights like those by Nakov and his team, the development of high-performing fake news detectors is an ever-evolving endeavor. “It’s an adversarial game,” Nakov said. “If I want to generate fake news, the moment I know how detection is done, I can give instructions to a generator to avoid detection.”

In the future, our textual milieu will continue to change, and writing may evolve into a deeper collaboration between humans and machines. Most research today makes a distinction between human-written and machine-generated text, Nakov said. But there are other gradations. A piece of text can be written by a human and polished by a machine. Or it can be written by a machine and humanized by a person.

While this added nuance related to the provenance of a piece of writing may prove to be even more challenging for fake news detectors, their ultimate goal will remain the same — to separate fact from fiction.