A model can land the right financial answer and botch every step to get there

Give a large language model a compound-interest problem and it will often hand back the correct dollar figure. Ask it to show its work, and the steps in between can fall apart: a skipped exponent, a percentage treated as a whole number, a rate divided by the wrong period. So even if the final number still checks out, the path to it often doesn’t.

This issue is the subject of FinChain, a new benchmark for financial reasoning built by researchers at MBZUAI and collaborators from institutions including Cornell, IIT Delhi, the University of Tokyo, and The Fin AI. The paper was accepted to ACL 2026, which is currently taking place in San Diego, California. Its premise is that in finance, getting the answer right isn’t worth much if you can’t trust how the model got there.

The authors took on this work because most financial benchmarks grade only the final answer. FinQA and ConvFinQA, the field’s standard datasets, treat reasoning as numerical question-answering and check whether the last figure matches. The authors argue that this can’t tell genuine multi-step reasoning from what they call “surface-level pattern matching.” A model that guesses its way to the right number looks, on those tests, exactly like a model that actually reasoned.

The fix the team came up with was to write every problem as a small program. Each template is parameterized Python: variables such as principal, interest rate, and time get sampled at random, and the same code that produces the question also computes the answer and every intermediate step. So the “gold” reasoning isn’t a human-written paragraph that might be sloppy; instead, it’s executable and can be checked line by line. Generating fresh numbers each time also keeps the test out of training data, which matters because most benchmarks leak. Once a dataset is public, it tends to end up in the next model’s training run, and scores stop meaning much. FinChain isn’t affected by this issue as it can mint new instances forever.

Figure 1: Symbolic template for generating compound interest problems in FinChain.

The benchmark spans 58 topics across 12 domains, from personal and corporate finance to crypto and sustainable finance. Each topic gets five templates that climb in difficulty, from two-step problems to roughly four-step ones such as adding a mid-term deposit or back-solved principal. Sampled 10 ways each, that comes to 2,900 test cases.

Figure 2: FinChain taxonomy of financial reasoning topics. Our benchmark spans 58 topics organized into 12 major domains, ranging from traditional areas like Corporate Finance and Financial Reporting to emerging fields such as Crypto Finance and Sustainable Finance. This hierarchical structure enables fine-grained evaluation of symbolic reasoning across diverse financial domains.

To score the models, the authors introduce ChainEval, a metric that compares a model’s chain of steps against the gold chain even when the two don’t line up neatly (because a model might split one step into three, or fold three into one.) To handle that, the authors borrow Dynamic Time Warping, a technique from speech recognition that aligns two sequences running at different speeds. Each matched pair of steps is scored twice: once for whether the words mean the same thing, once for whether the numbers agree. A step earns credit only when both line up. The final grade blends that step-by-step alignment with plain final-answer correctness.

The tolerance for “the numbers agree” is set at 5% because in financial auditing, deviations under that threshold are treated as immaterial. The authors also tested a wide range and 5% happened to track human judgments best, with the metric agreeing with expert ratings far more closely than older text-overlap scores such as ROUGE.

Run across 26 models, the results show frontier systems coming out on top, but barely, and not cleanly. GPT-5-mini (67.17) edged out full GPT-5 (66.57). Two small fine-tuned 7-billion-parameter models (Mathstral at 59.87 and the finance-tuned Fin-R1 at 58.14) landed within reach of frontier systems many times their size. And one math-specialized model, MetaMath, scored 7.93, close to the floor. In other words, bigger did not reliably mean better.

Several models posted much higher final-answer scores than reasoning scores. As the paper puts it, “correct final answers do not necessarily imply faithful intermediate reasoning.” A model can arrive at the destination while the route is nonsense.

Every model (including GPT-5 and Gemini 2.5 Pro) degraded on the hardest problems, which required around four chained steps. But while the frontier systems slide, the smaller ones fall off a cliff. Completing a long reasoning chain to the end, the authors find, is the real bottleneck.

On a hand-checked set of 200 questions, GPT-5-mini got 80.5% fully correct, and its remaining mistakes were mostly ordinary arithmetic slips and concept errors. Finance-Qwen, a finance-tuned model, erred on 90% of the same questions and its failures were messier, including invented numbers and output so malformed it couldn’t be parsed into steps at all. Stronger models fail in a few predictable ways; weaker ones fail in every direction at once.

Every problem is synthetic, spun from a template, so the benchmark sidesteps the noise of real filings, earnings calls, and regulatory prose. It’s English-only and built around U.S. financial conventions. And the scoring depends on chopping a model’s free-form answer into discrete steps, a step that’s sensitive to formatting quirks. It’s important to remember that FinChain measures whether a model can reason cleanly under controlled conditions, not whether it can do an analyst’s actual job.

That said, ten financial experts such as industry quants and finance graduate students reviewed all 290 templates to ensure anything that was incorrect could be revised before the experiments were run. Most flagged problems were about how numbers and units were displayed, not broken logic.

According to Zhuohan Xie, “In finance, getting the final number right is not enough. The assumptions, formulas, and intermediate steps also need to be auditable, because a small hidden error can change the decision. With FinChain, we wanted to move evaluation from ‘did the model guess the right answer?’ to ‘can we verify the reasoning that led there?'”

The paper is a great reminder why systems being folded into trading desks, audit workflows, and personal-finance tools need more auditing: they can confidently produce a figure even when the reasoning beneath it is broken. Until now, the tests waving them through were grading only the figure. FinChain gives us a mechanism to test the underlying intelligence too.