Skip to content
NLEN
Illustration: Evaluating reasoning steps in math and logic tasks

Evaluating reasoning steps in math and logic tasks

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

When testing large language models, math and logic tasks have traditionally served as the litmus test for analytical ability. Collections such as GSM8K (Grade School Math) and MATH are widely used to measure how well a model correctly parses and solves multi-step problems. In practice, however, a binary assessment — where only the final outcome is marked right or wrong — turns out to be far too blunt an instrument. This approach doesn't reveal why a model fails or succeeds, leading to a distorted picture of its actual reasoning quality.

To make reliable statements about a model's suitability for complex business processes, financial calculations, or logical planning problems, evaluating the entire reasoning path is essential. In this article we cover the methodological pitfalls of single-answer evaluation, the need for standardized normalization, a detailed error taxonomy, the role of multiple attempts, and how to score intermediate steps in an automated yet calibrated way.

The core problem with final-answer evaluation

Evaluating math and reasoning tasks based solely on the final result introduces two major sources of noise into a test setup: false positives and false negatives.

A false positive occurs when a model reaches the correct final number through flawed reasoning. Think of a situation where an arithmetic error in the second intermediate step happens to be offset by a second arithmetic error or a misapplied formula in the fourth step. It also happens that a model simply repeats or guesses a number from the question and coincidentally hits the expected value. Anyone who looks only at the outcome rewards a defective reasoning process in these cases.

Conversely, a false negative leads to underestimating the model's capabilities. When a model produces a flawless logical derivation and correct formulas across seven steps but makes a typo or shows a rounding difference in the very last step, the entire response gets a score of zero. In production environments, however, the first seven steps of logic can be extremely valuable, while the arithmetic error could have been caught with a simple calculator tool or validation step.

Key insight: A correct final answer doesn't prove the reasoning was sound, and an incorrect final answer doesn't reveal where the process went off track. Anyone who wants to understand the quality of reasoning models must separate the intermediate steps from the final outcome.

Answer accuracy versus step accuracy

To make reasoning quality visible, we use two separate metrics: outcome accuracy and process accuracy. Each metric provides guidance for a different type of decision.

Outcome accuracy

Outcome accuracy measures the percentage of problems where the final result, after any normalization, exactly matches the reference value. This metric is useful as a quick initial triage of models and indicates the extent to which a system produces directly correct values without human intervention. However, it says little about the model's robustness when input values change slightly.

Process accuracy

Process accuracy measures the logical and arithmetic correctness of each individual intermediate step in the reasoning chain. This involves determining whether step n logically and mathematically follows from the given information and the preceding steps 1 through n-1. The use of chain-of-thought techniques makes these intermediate steps explicitly visible, making process accuracy measurable.

Metric What it measures Suitable for decisions about
Outcome accuracy Match between the final result and the reference. Suitability for direct customer contact or automated processing without review.
Process accuracy Validity of individual reasoning steps. Diagnosing model quality, selection for fine-tuning, and integration with external tools.

When a model has high outcome accuracy but low process accuracy, the system is vulnerable to unexpected failures with small variations in the question. High process accuracy combined with occasional arithmetic errors, on the other hand, points to a strong reasoning foundation that simply needs support from a calculator plugin or Python interpreter.

Making answers comparable: the need for normalization

A common source of noise when evaluating outcome accuracy is the absence of a strict normalization workflow. A language model generates free-form text. If the reference value 42 , the model might answer with 42, 42.0, € 42,-, 42 km/h or Totaal: 42. Without prior cleanup, the evaluation tool measures formatting preferences instead of mathematical skill.

A robust normalization pipeline consists of four consecutive stages:

  1. Isolating the core value: Applying regular expressions or parsing instructions to extract the final value from an answer block (for example, from the well-known \boxed{...} notation used in MATH benchmarks).
  2. Stripping units and currency: Removing textual additions such as currency symbols, percent signs, or units of measurement, provided the problem doesn't explicitly ask for a specific unit conversion.
  3. Harmonizing number notation: Converting fractions (such as 1/2) to standard decimals (0.5), replacing commas with periods for decimal separators, and removing thousands separators.
  4. Setting tolerance and rounding: Applying a relative or absolute margin of error (epsilon) for floating-point numbers. An answer such as 3.1415 should not be counted as incorrect relative to 3.14.

Only once two answers have passed through this pipeline can a fair comparison be made. Without this step, models that reason correctly in substance are unfairly penalized for their syntactic style.

Types of errors worth distinguishing

When an intermediate step or final answer is incorrect, a simple “wrong” label provides insufficient guidance. To make targeted improvements to prompts, fine-tuning datasets, or system architecture, it is necessary to categorize errors into four main categories.

1. Comprehension error (misunderstood problem)

The model misinterprets the initial situation or the conditions of the problem. For example, it ignores an explicitly stated constraint or swaps variables in the problem description. The reasoning that follows may be internally logical but is based on the wrong starting point. The solution here usually lies in better instructions or clearer contextualization in the prompt.

2. Arithmetic error in an otherwise correct approach

The model chooses the right logical route and sets up the correct equations but makes an arithmetic error during execution (for example, an addition error or a mistake multiplying two numbers). This is a well-known phenomenon in pure language models. The solution for this type of error is to bring in external tools, such as a Python environment or a calculation module via tool calling.

3. Strategic derailment

The model starts correctly and executes the first steps properly but takes a wrong turn partway through. For example, it applies a mathematical rule that isn't valid in this specific context, or forgets to carry an intermediate result forward to the next step. This points to a lack of deep logical insight and can be addressed with targeted training material or multi-step checks.

4. Unfounded conclusion (hallucination of the final step)

The intermediate steps and logical structure are present, but the final answer doesn't follow from the preceding reasoning in any way. The model “jumps” to an arbitrary value. This indicates a disruption in the generation process and can often be reduced by lowering the temperature setting or enforcing structured output format requirements.

Avoiding contamination with your own variants

Public collections such as GSM8K and MATH have been available for years and have been widely incorporated into the training data of modern language models. This leads to data contamination: the model doesn't generate the answer based on reasoning ability but on pattern recognition or recalling the exact text from the training corpus.

For an in-depth analysis of the general mechanisms behind this problem, see our analysis of the phenomenon of benchmark contamination. Specifically for math and logic tasks, contamination is best avoided by constructing your own variants based on existing problems. Three methods are especially effective for this:

Multiple attempts and variance in reasoning behavior

Language models generate text based on probabilistic sampling. When the temperature setting is above zero, the same problem produces different outcomes and reasoning paths across multiple runs. Judging a model based on just a single attempt per problem therefore gives an unreliable picture.

To determine the reliability of a reasoning model, we use multiple samples per problem. Two evaluation metrics are key here:

Pass@k

The pass@k metric indicates the percentage of problems where the model, in at least one of the k generated attempts, finds the correct answer. This measure is valuable when the system is deployed in an environment where multiple answers may be generated in parallel and an external verifier selects the best option.

Spread and stability

By comparing the outcomes of 10 or 20 attempts per problem, the model's internal variability becomes visible. If a model reasons in exactly the same way and reaches the same answer in 9 out of 10 attempts, the reasoning path is stable. If the model produces 5 different final answers via 5 substantively different lines of reasoning across 10 attempts, the system is unreliable — even if 2 of those attempts happen to contain the correct answer.

This approach closely aligns with advanced prompting methods, such as self-consistency prompting for reasoning steps, where the most frequent line of reasoning is chosen as the definitive answer. For the statistical basis behind the required sample size for this type of measurement, see the guide on statistics for evaluations.

Evaluating intermediate steps with a judge model

Manually evaluating hundreds of reasoning steps per problem is practically infeasible for large benchmarks. A pragmatic method is to use a powerful language model as an automated evaluator (LLM-as-a-Judge) to check the intermediate steps.

In automated step evaluation, the judge model is given the following information for each step:

  1. The original problem and the stated constraints.
  2. The steps certified as correct so far, 1 through i-1.
  3. The step being evaluated, i.

The judge model is given the specific task of determining whether step i logically and arithmetically follows from the preceding context. If not, the judge must indicate what type of error was made.

Warning about automated evaluation: A judge model can make mistakes of its own. It sometimes tends to score detailed, lengthy answers higher than short, direct proofs, or it overlooks subtle arithmetic errors. A judge model must therefore always be calibrated first.

Calibrating the automated judge is done by having human experts manually evaluate a representative sample of reasoning steps (for example, 200 steps). You then compare the automated judge's verdicts with the human annotations. Only once the precision and recall of the automated judge are sufficiently high on this validation set can the evaluation process be automated at scale.

What this means for your own application

The key lesson for organizations and developers is that public leaderboards based on GSM8K or MATH provide, at best, an initial indication of general capabilities. They don't predict how a model will perform on the specific logic, calculation rules, and constraints of your own domain.

For a comparison of the fundamental reasoning architectures of modern models, see the comparisons between reasoning models. To make performance concrete for your own organization, it's advisable not to rely on public benchmarks but to build your own internal evaluation set.

An effective approach for your own evaluation set includes the following steps:

For a detailed step-by-step plan on setting up such a test environment, see the step-by-step plan for setting up your own benchmark.

Further reading