Verifying language model reasoning from the inside out

Test-time compute scaling enhances large language model (LLM) performance by granting the model more time to explore multiple reasoning paths. This approach effectively trades increased GPU compute for higher-quality outputs.

An essential component of test-time compute scaling is evaluating the quality of reasoning paths, or even individual reasoning steps, in order to guide the model toward more promising directions.

One way to perform this assessment is to use a separate critic model, known as a process reward model, or PRM. A PRM is typically another language model that runs alongside the main model and evaluates the quality of its reasoning traces. While PRMs can be effective, they are often large models themselves, sometimes with billions of parameters, and can, therefore, substantially increase computational cost.

Researchers from MBZUAI, ETH Zurich, and the National University of Singapore have developed a new verification approach called ReProbe. Instead of relying on a PRM for verification, ReProbe analyzes the main model’s own internal states to determine how confident the model is in its reasoning. The idea is that the more confident a model is as it reasons, the more likely it is to be right.

“Think of ReProbe as a kind of lie detector for a language model,” says Artem Shelmanov, assistant professor of practice in Natural Language Processing at MBZUAI. “If PRMs evaluate the textual answers produced by an LLM, ReProbe is like placing electrodes directly on the model’s ‘brain’ to detect whether it is hesitating.”

Shelmanov is a leader of the ReProbe development team and an author of another work on Uncertainty Quantification Heads, a framework for analyzing the internal states of LLMs that was originally used for hallucination detection, that laid the foundation for this approach.

The work on ReProbe will be presented in an oral session at the 64th Annual Meeting of the Association for Computational Linguistics (ACL). Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, and Mrinmaya Sachan are co-authors of the study.

How ReProbe works

The researchers describe ReProbe as a “plug-and-play module” that sits on top of a base LLM, outputting a probability of how likely the current reasoning step is to be correct. It does this by extracting features from the base model’s internal states for each token it generates – either attention weights and logits, or hidden states from the model’s layers.

In the previous study on Uncertainty Quantification Heads, Shelmanov and others explored a similar approach to detect language model hallucinations, but this is the first time it has been used to analyze reasoning ability.

“We’re trying to extract as much information as possible from the model itself, because when you use external tools such as PRMs or knowledge bases, that external knowledge may be messy or incomplete, and the overhead of employing it can be large,” Shelmanov says.

For example, a PRM can add up to 8 billion parameters, leading to significant computational costs. PRMs are also domain-specific and don’t generalize well to other domains. ReProbe has fewer than 10 million parameters and shows strong performance across domains.

Training ReProbe

The researchers trained ReProbe so that it could identify what a confident, or correct, internal state looks like. To build the training dataset, they had a base language model reason through more than 10,000 math problems from a dataset called PRM800K. This generated more than 30,000 reasoning trajectories. Each step was then labeled as correct or incorrect by a language model acting as a “judge.” The researchers did this in a supervised setup, where DeepSeek-R1 acted as the judge, and a self-supervised setup, where the base model evaluated its own responses. ReProbe can be trained in a streamlined way, without any external verifier or human-annotated data, and the base language model isn’t modified during training.

What they found

The researchers compared ReProbe in several configurations against PRMs of different sizes. For the base language models, they used two state-of-the-art LLMs, Qwen-3 and Phi-4. The PRMs ranged in size from 1.5 billion to 8 billion parameters.

They tested the model configurations on three different domains: mathematical reasoning, planning, and general knowledge question answering. Both ReProbe and the PRMs were trained only on math problems, so mathematical reasoning was considered in-domain, while planning and general knowledge question answering were considered out of domain. The researchers also evaluated three settings: step-level correctness, best-of-N, and beam search.

In the step-level correctness setting, the ReProbe variants matched or exceeded the performance of even the largest PRMs on average, though the top PRMs retained a slight advantage on the math problems that both were trained on. Even so, the result is impressive, Shelmanov says, considering that the large PRMs are 810 times the size of ReProbe. And beyond matching the accuracy of PRMs, ReProbe was also significantly faster, running 2.6 to 25 times faster than the PRMs it was tested against in beam-search decoding.

On average, ReProbe performed better than PRMs on out-of-domain data. Shelmanov says that “this is because our approach is based on confidence, and confidence is something that is more or less universal across domains.”

But the choice doesn’t necessarily need to be between ReProbe and PRMs, Shelmanov and his co-authors write. They studied performance by combining the two approaches and saw improvement compared to either approach alone.

This suggests, they say, that the methods capture different aspects of reasoning quality, pointing “to a promising path toward hybrid verifiers that combine introspective confidence with process rewards.”