Search⌘ K
AI Features
Home
Courses
LLM Bootcamp

BLEU, ROUGE, and METEOR

Explore BLEU, ROUGE, and METEOR evaluation metrics used for assessing large language model outputs. Understand how each metric measures precision, recall, or semantic relevance and when to apply them in translation or summarization tasks. Gain insights into their strengths and limitations to interpret evaluation results effectively and improve production LLM pipelines.

We'll cover the following...

When an LLM generates a translation or a summary, there is no single “correct” answer. A sentence like “The cat sat on the mat” can be validly translated or summarized in dozens of ways. This fundamental ambiguity makes classification-style accuracy useless for evaluating generation tasks. Instead, the field relies on reference-based evaluation, where the model’s output (called the candidate) is compared against one or more human-written reference texts. Three metrics have dominated this space for decades: BLEU, ROUGE, and METEOR. Each one measures overlap between candidate and reference from a different angle, and understanding their mechanics is essential for interpreting evaluation results in any production LLM pipeline. Tools like Amazon SageMaker provide built-in support for all three, making them practical choices beyond academic benchmarks.

This lesson walks through how each metric works, when to use it, and where it falls short.

BLEU: precision-oriented n-gram matching

How BLEU computes its score

BLEU (Bilingual Evaluation Understudy) remains the most widely reported metric in translation research.

The core mechanism is straightforward. BLEU counts how many n-gramsContiguous sequences of n words; a unigram is a single word, a bigram is two consecutive words, and so on. in the candidate appear somewhere in the reference. This makes it a precision-oriented metric because it asks “of everything the model said, how much was correct?” rather than “how much of the reference did the model cover?”

Standard BLEU (often called BLEU-4) combines precision scores across four n-gram levels using a geometric mean.

  • Unigram precision: Measures individual word overlap, capturing vocabulary accuracy.

  • Bigram precision: Measures two-word phrase overlap, capturing basic word pairing.

  • Trigram and 4-gram precision: Measure longer phrase overlap, capturing fluency ...