Skip to content
NLEN
Illustration: Evaluating the translation quality of LLM output

Evaluating Translation Quality in LLMs: COMET, BLEURT & BLEU

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

Quantifying translation quality is one of the most complex challenges in natural language processing. Where classic classification tasks or arithmetic tests have a single unambiguous right or wrong answer, a translation has a virtually infinite range of correct outcomes. With the rise of large language models (LLMs) as a replacement for, or addition to, traditional neural machine translation (NMT) systems, the need to measure translation quality accurately, reproducibly, and at scale has only grown.

In this article, we break down how automated translation quality evaluation methods work. We cover the evolution of evaluation technology, analyze the mathematical and conceptual limitations of classic overlap metrics, and look at how modern neural metrics such as COMET and BLEURT have changed the evaluation landscape. Finally, we discuss the specific pitfalls in evaluating Dutch and offer a practical framework for setting up your own evaluation pipeline.

The fundamental problem of translation evaluation

Measuring translation quality runs into a linguistic fact: equipollence, or functional equivalence. A single source sentence in a language such as English can be correctly translated into Dutch in dozens of ways. The choice of synonyms, a different sentence structure, passive versus active voice, or a variation in word order can all produce perfectly natural and accurate translations, even though the lexical elements used differ substantially.

Take, for example, the English source sentence: "The company decided to postpone the product launch due to unexpected software bugs." Possible correct Dutch translations include:

If an automated evaluation system strictly compares the generated translation to only one of these variants as the reference, valid synonyms ("onderneming" versus "bedrijf") and alternative but grammatically correct constructions get penalized. An automated metric must therefore be able to recognize semantic equivalence independent of surface-level word overlap.

The three generations of evaluation methods

Three clear generations can be distinguished in the development of automatic translation evaluation. Each generation introduced a new paradigm to narrow the gap between machine assessment and human perception.

Generation Methodology Examples Key advantage Main limitation
1. Overlap metrics Exact n-gram comparison at the word or character level BLEU, ROUGE, METEOR, chrF Very fast, cheap, deterministic Doesn't account for synonyms and sentence structure; depends on an exact reference
2. Learned neural metrics Neural networks (BERT-based) trained on human ratings COMET, BLEURT High correlation with human judgments; understands context and synonyms Black-box behavior; requires GPU compute; sensitive to specific training data
3. LLM-based evaluation Prompting generative language models with error rubrics Gemini, GPT-4, Llama 3 with MQM prompts Detailed qualitative feedback and error analysis per category High cost per evaluation; inconsistency; risk that the model favors its own output

First generation: word-level overlap metrics

The first generation of automated metrics focused on the statistical overlap between the generated translation (the hypothesis) and one or more human-written reference translations. The best-known representative is BLEU (Bilingual Evaluation Understudy), but metrics such as ROUGE and chrF also belong to this category.

These metrics count the number of overlapping n-grams (consecutive sequences of n words or characters) between the hypothesis and the reference. Because they look only at exact textual overlap, they overlook semantic relatedness. Sentences with exactly the same meaning but different word choices get a low score, while sentences with a lot of overlapping words but an incorrect negation can score high.

Second generation: learned neural metrics

To get around the rigidity of exact n-gram overlap, the second generation introduced machine-learned evaluation models. Metrics such as BLEURT and COMET use pre-trained multilingual transformer models (such as mBERT or XLM-RoBERTa) that are then fine-tuned on large datasets of human quality judgments, such as the Direct Assessment (DA) or Multidimensional Quality Metrics (MQM) datasets from the annual WMT evaluation competitions.

These models convert the source text, the hypothesis, and the reference into dense vector representations (embeddings). This means they don't evaluate whether the words literally match, but whether the meaning and contextual gradients are close together in vector space. This allows them to correctly value synonyms, paraphrases, and structural rewordings.

Third generation: evaluation by generative language models

The most recent development is using large generative language models as evaluators. Instead of returning a compressed floating-point number as a score, the LLM evaluator reads the source text and the translation and applies a structured error rubric to it. This allows the evaluator not only to give a quality score, but also to flag and categorize specific segments (for example, critical translation errors, style errors, or terminology deviations).

Overlap metrics analyzed in depth: BLEU and chrF

Despite their well-known shortcomings, overlap metrics remain widely used in scientific literature and industrial benchmark pipelines. It's therefore essential to understand how they work and where the limits of their applicability lie.

How BLEU works

BLEU calculates the modified n-gram precision for $n=1$ through $n=4$. Precision calculates what percentage of the n-grams in the hypothesis also appear in the reference text. To prevent a hypothesis that simply repeats one correct word ten times from getting a perfect score, BLEU uses 'clipped' precision: an n-gram cannot be counted more times than the maximum number of times it appears in a single reference.

Because precision alone would allow a translation of just two words to achieve a score of 100% as long as those two words appear in the reference, BLEU includes a length penalty (the Brevity Penalty or BP). The final formula is:

BLEU = BP × exp( Σn=1..4 wn log pn )

Where pn represents the n-gram precision and wn the weights (usually distributed evenly as 0.25). The length penalty BP is calculated as:

BP = min(1, exp(1 − r / c))

Here, c is the length of the hypothesis and r the effective reference length. When the generated sentence is shorter than the reference, the total score is reduced exponentially.

Why BLEU fails at the sentence level

BLEU was designed as a metric at the corpus level. When BLEU is applied to a single isolated sentence (segment level), a mathematical problem arises. If there's no exact match of four consecutive words between the hypothesis and the reference for $n=4$, this becomes p4 = 0. Because the formula calculates a geometric mean via logarithms, this immediately drops the entire BLEU score for that sentence to zero, regardless of how well the individual words were chosen.

Although smoothing functions exist to work around this problem, the correlation between BLEU scores and human judgments at the sentence level remains very low. In addition, BLEU rewards systems that follow the source language's grammar too closely when the reference translation is itself a somewhat literal translation, while penalizing more creative but correct translations.

The importance of standardized preprocessing: SacreBLEU

A historical problem with BLEU scores in scientific literature was their sensitivity to tokenization and text preprocessing. Changes in how punctuation is separated, how capitalization is handled, or how compound words are split can cause the BLEU score to vary by several points. As a result, scores across different scientific papers were often not directly comparable.

To solve this, SacreBLEU was developed. SacreBLEU automates downloading standard test sets and applies a standardized, immutable tokenization. When reporting BLEU scores, using SacreBLEU is the norm, and the exact parameter signature (such as nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1) should always be included.

chrF: character level as an alternative

A strong alternative within the first generation is chrF (character n-gram F-score). Instead of words, chrF compares character n-grams (usually $n=6$). This offers major advantages for morphologically rich languages and languages with many compound words, such as Dutch and German. Because chrF looks at overlapping character sequences, small variations in inflections, conjugations, and plural forms aren't immediately counted as a complete error. Various WMT evaluations show that chrF has a significantly higher correlation with human judgments than BLEU.

Learned neural metrics: COMET and BLEURT

To overcome the limitations of n-gram overlap, modern evaluation pipelines make extensive use of neural evaluation models. The two most prominent frameworks in this domain are COMET and BLEURT.

BLEURT: fine-tuning on synthetic and human data

BLEURT (Bilingual Evaluation Understudy with Representations from Transformers) was developed by Google. The model starts with a pre-trained multilingual BERT model. The crucial step in BLEURT's development is the intermediate training phase using synthetic data.

Before the model is trained on scarce human ratings, millions of perturbed sentences are generated. This is done by removing words from existing Wikipedia sentences, replacing them with synonyms, changing the word order, or adding machine translations. The model learns to predict which changes were made and calculates simulated scores for, among other things, BLEU and BERTScore. Only after this phase is the model fine-tuned on real human quality ratings (Direct Assessment scores). This allows BLEURT to excel at assessing subtle grammatical-semantic deviations.

COMET: modeling with or without a reference

COMET (Crosslingual Optimized Metric for Evaluation of Translation), developed by Unbabel, uses an architecture based on XLM-RoBERTa. COMET converts the source sentence, the generated hypothesis, and the reference translation into vector representations. The model then combines these vectors through element-wise absolute differences and multiplications, after which a multi-layer neural network (a feed-forward neural network) calculates a final quality score.

A fundamental conceptual distinction within COMET is the choice between reference-based and reference-free models:

Note: The choice between a reference-based or reference-free metric determines exactly what you're measuring. Reference-based models test whether the system can approximate a specific target translation, while reference-free models (QE) primarily measure whether the translation is semantically faithful to the source and reads fluently in the target language, independent of specific human phrasings.

Evaluation by a language model with an error-focused rubric

The newest development in evaluation methodology is deploying generative LLMs as evaluators. Here, the language model is asked to analyze the source text and the generated translation and categorize errors according to a structured framework, such as the MQM (Multidimensional Quality Metrics) framework.

In such a setup, the evaluation model is tasked with sorting errors into specific categories, including:

Each detected error is assigned a severity level (for example, minor, major, or critical), after which a formula calculates the final quality score. This process yields extensive diagnostic information that can't be obtained from a single BLEU or COMET number. More on how these evaluation forms are set up can be found in the overview of LLM-as-a-judge methods.

There is, however, an important methodological risk with this approach: the self-enhancement bias , or the 'player and referee' problem. When a specific LLM family member (such as a GPT or Claude variant) is used to judge translations generated by the same or a related model, the judging model often assigns a higher score to its own style, sentence constructions, and word choice. To prevent this, the evaluation model must be kept strictly separate from the generation model being tested, or the evaluation must be cross-validated with neutral neural metrics.

Specific challenges in evaluating Dutch

Evaluating Dutch translations with automated models introduces specific linguistic complications. General-purpose metrics primarily optimized on English-centric data regularly fall short on these points.

1. Forms of address and register

English makes no grammatical distinction between the formal and informal second-person singular ("you"). In Dutch, the difference between "u" and "je/jij" (and the corresponding possessive pronouns "uw" versus "jouw/je") is crucial to a text's tone. An automated metric such as BLEU will heavily penalize a translation using "u" if the reference uses "je," even though both are grammatically correct. Neural metrics such as COMET understand this synchronicity better, but can sometimes overlook inconsistencies within a single document (mixing up u and je) if the source language makes no such distinction. For more background on language-specific characteristics, see the article on language diversity in LLMs for Dutch.

2. Compounds and closed-form spelling

Dutch is a language that writes compound words as one word (for example, "kwaliteitsbeoordelingssysteem"). Many tokenizers or translation models carried over from English tend toward what's sometimes called English disease: incorrectly splitting compound words apart ("kwaliteits beoordelings systeem"). Oddly enough, word-level overlap metrics sometimes penalize incorrect compound splitting less heavily when the separate words match the source, while character-based chrF and trained COMET models do correctly detect and penalize these errors.

3. Belgian Dutch versus Netherlandic Dutch

Within the Dutch-language area, there are clear lexical and grammatical differences between the variants used in the Netherlands and in Flanders (for example, "geheel vrijblijvend" versus "reeds", or "magnetron" versus "microgolfoven"). If a test set contains references in Netherlandic Dutch, a model that produces excellent Belgian Dutch will be scored lower by standard overlap metrics, despite flawless quality. This is a critical variable when setting up tests for specific markets; read more about this in the guide on testing Dutch-language models.

4. 'Translationese' and natural language use

A common phenomenon in machine translations and LLM generations is grammatically correct but unnaturally sounding text (also called translationese or 'translation flavor'). The model follows the syntactic structures of the English source text too closely. Neural metrics trained on human judgments often score texts with translationese surprisingly high, because the semantic content is exactly right and the sentence structure isn't outright wrong. Only human sampling or advanced LLM rubrics that specifically test for idiomatic naturalness reveal these shortcomings.

Segment level versus corpus level

A crucial aspect of interpreting evaluation scores is the level at which aggregation takes place. A common mistake in evaluation architectures is relying solely on a single aggregated final score across the entire dataset.

When a test corpus consists of 1,000 sentences, a model might deliver an excellent translation on 990 sentences but make a critical error on 10 sentences (for example, omitting the word "not," which exactly reverses the meaning of the sentence). In an aggregated corpus score for BLEU or COMET, these 10 critical errors disappear completely into the average. The model's overall score remains comfortably sufficient, while the system is unreliable for a production environment.

For a valid evaluation, scores must be analyzed at two levels:

  1. Corpus level: For comparing the overall performance of two model versions or different systems.
  2. Segment level: For identifying outliers. By filtering for sentences with the lowest individual COMET or BLEURT scores, you directly trace critical system errors, hallucinations, or model failures.

A workable setup for your own evaluation pipeline

To reliably evaluate the translation quality of an LLM or NMT system within an organization, it's wise to follow a structured step-by-step plan. A robust evaluation doesn't require thousands of sentences, but it does require a well-thought-out methodology.

Step 1: Assemble a domain-specific test set

Don't use generic public datasets (such as Flores or WMT) as your primary benchmark; LLMs have often already been trained on this data (data contamination). Put together your own, protected test set of 100 to 300 segments representative of your specific application domain (for example, legal contracts, technical manuals, or marketing copy). Make sure you have high-quality reference translations produced by professional human translators.

Step 2: Implement two independent automated metrics

Always combine a classic or fast metric with an advanced neural metric. A proven combination is:

Step 3: Run a periodic human sample check as a calibration point

Automated metrics are tools, not absolute truth. Take a random sample of 10 to 20% of the evaluated segments and have them assessed by a native speaker or domain expert. Have the reviewer assign scores or flag errors using a simplified MQM rubric. Periodically compare the human judgments with the COMET scores to verify that the automated metric still correlates with actual perceived quality. For setting up such panels, see the guide on human evaluation of language models.

What do you do with the results?

Collecting evaluation scores isn't an end in itself. The outcomes from the evaluation pipeline should drive further optimization of the language task:

Model selection and cost trade-offs

With quantitative scores, you can make informed choices about which model to deploy. If a smaller, cheaper open-source model achieves a COMET score on your specific test set that barely differs from an expensive, commercial model, you can confidently switch. You'll find detailed characteristics of different models in the knowledge base on models for translation tasks.

Prompt engineering and steering instructions

When segment analyses or LLM error rubrics show that the model consistently makes mistakes with forms of address (for example, randomly switching between "u" and "je"), you can adjust the system prompt in a targeted way. Prompt changes can be run straight back through the test set to measure the effect on the score. More techniques for multilingual instructions can be found in the overview on multilingual prompting strategies.

Building a dynamic terminology list (glossary)

If the evaluation shows that specific jargon terms are being translated incorrectly or inconsistently, adding a fixed reference file or glossary helps. By feeding the model specific preferred translations through in-context learning or a glossary pipeline, terminology errors are eliminated immediately without having to retrain the entire model.

Further reading