Separating fact from fiction with uncertainty quantification

You may have heard about the lawyers in New York who were fined for submitting a court filing that contained references to fictional rulings. It wasn’t their fault, they said. The lawyers had used ChatGPT to help them write the document and weren’t aware at the time that the chatbot, in addition to its impressive ability to fluidly generate text, also has an impressive ability to make things up.

This phenomenon is known as hallucination and it is a problem that has limited the usefulness of language models to date.

Maxim Panov, assistant professor of machine learning at Mohamed Bin Zayed University of Artificial Intelligence, is developing methods that are grounded in theoretical statistics that can be used to improve the utility of language models and other machine learning applications.

Predictive machines

Maxim Panov

Panov’s doctoral work related to theoretical statistics, and he received a master’s degree in applied mathematics with a focus on machine learning. “My doctoral studies provided the grounding for my current style of research, where I’m not interested in only maximizing the quality of a prediction, but to ground my work in theory,” he said.

He is interested in what’s known as uncertainty quantification for machine learning models. As its name suggests, uncertainty quantification is a method that provides insight into the level of confidence a machine-learning model has about a prediction it makes.

Machine-learning models — which are today often built on neural networks — are, in a sense, predictive machines. These systems determine an output based on data they have encountered in the past and the ways in which they have been trained. Because the training data are limited and there is always inherent randomness in it, the resulting models are never perfect and there is always some level of uncertainty related to the predictions they make.

An added challenge is that unless they are programmed to do otherwise, machine-learning models will make a prediction one way or another — even if it’s a bad one. The hallucinations of language models are cases in which systems may not have high confidence in the text they generate but do so anyway.

Models that can abstain from prediction are particularly valuable in a field like medicine, where accuracy has real consequences for people. For other applications, such as chatbots, it may be enough for a model to notify a user that it isn’t so sure about its prediction.

Scientists can design a model from the start to communicate with users about their level of uncertainty. But Panov is working on post-processing techniques for uncertainty quantification, which are complements to machine-learning models that have already been trained.

“Post-processing is very important for modern applications because people tend to use very large neural networks, and retraining these networks is very expensive,” Panov said. “If we can develop tools that can be applied on top of existing neural networks, that’s something that would provide a lot of benefits to users.”

Post-processing techniques are also valuable because it’s not always known during the training phase how a machine-learning system will be used once it is deployed. “We want to improve neural networks’ ability to be applied to things for which they were not specifically designed,” Panov said.

Addressing hallucination

One example of Panov’s recent work is a study he authored with scientists from MBZUAI and other institutions that proposes an approach for identifying language models hallucinations. The work addresses a significant need, as most models don’t currently identify information in their outputs that may be unreliable.

“So many people are using language models for a wide variety of tasks, and yet, even for the best models, the amount of wrong information that is produced is immense,” Panov said. “We want to try to identify incorrect claims based on the methods of post-processing, while doing so in a way that is statistically and theoretically grounded.”

The problem is not only that language models hallucinate, but that incorrect or completely made-up information can appear in the same sentence as accurate statements, making it difficult for people using these systems to separate fact from fiction.

“When you read what a language model produces, there’s no way to distinguish what is correct from what isn’t,” Panov said. “If we can identify a significant number of cases where a model is unsure of its output and highlight them for users, that would result in a huge improvement to the trustworthiness of these systems.”

Panov and his colleagues’ approach essentially makes it visible to users when a language model is uncertain about the text it produces. They describe their approach as “claim conditioned probability” uncertainty quantification, and it is based on an open-source language model’s own methods for constructing sentences via probabilistic reasoning on a token level.

A token in natural-language processing is essentially a word or part of a word that is processed by a computer. When a language model is generating an output, it predicts what the most appropriate string of tokens will be based on context. Some tokens in a series appear with high certainty, others with lower certainty.

Panov and his colleagues’ method identifies tokens that the model might not be so sure are correct in a particular sequence. They can measure the uncertainty of tokens in a sequence and use these calculations to determine the level of uncertainty of claims that are made by a language model. As they write in their study, “uncertainty is relevant for fact-checking, because if the model is not sure about the information it relays, there might be a high chance of a factual mistake.”

New directions

There are certainly improvements that can be made to systems like ChatGPT to increase their accuracy. But no system will ever perform perfectly all of the time. “Ideally, the goal is to create a model that never makes mistakes,” Panov said. “But any model that is developed will have issues, and we need to have methods for solving them.”

When considering future directions for his research, Panov considered how uncertainty quantification may be included in systems from the start. “I’m interested to develop architectures and learning procedures that on the one hand work well, but on the other hand, will have this uncertainty quantification capability naturally embedded in them,” he said.