BLEU Score

Compute BLEU with brevity penalty against references.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

Enter a hypothesis (the machine output) and one or more reference sentences, one per line, and get a BLEU score with the four n-gram precisions and the brevity penalty. The text is lowercased and punctuation is removed before comparing.

How to use

1
Step 1
Enter the hypothesis.
2
Step 2
Enter the references, one per line.
3
Step 3
Press Calculate.

How it works

BLEU is the geometric mean of the clipped (modified) n-gram precisions for n = 1 to 4, multiplied by a brevity penalty that punishes outputs shorter than the reference. A zero precision is floored at 1e-9 instead of giving exactly 0, which is why a short sentence with no matching 4-gram scores a tiny non-zero value. With several references it scores against each and keeps the highest.

BLEU = BP · exp( (1/4) Σₙ ln pₙ )     BP = min(1, e^(1 − r/c))

Worked example

Cat on the mat

Input
Hypothesis: the cat sat on the mat Reference: the cat is on the mat
Output
BLEU 0.003344; P1 0.833, P2 0.6, P3 0.25, P4 0; brevity penalty 1
Walkthrough
Five of six words match (P1 = 5/6), but no 4-gram matches, so P4 is 0 and drags the geometric mean down. An identical hypothesis and reference give 1.

Limitations

  • Sentence-level BLEU is noisy; it is designed for corpus-level evaluation.
  • No smoothing method other than the 1e-9 floor, so short sentences score near zero.
  • Not the same implementation as sacreBLEU, so numbers will not match published results.
  • Case and punctuation are ignored. Tokens are split on whitespace after cleaning.

Questions

How do I calculate a BLEU score?
Enter the hypothesis and references and press Calculate.
Why is my BLEU so low for a short sentence?
If no 4-gram matches, P4 is 0, which sends the geometric mean to nearly zero. Sentence-level BLEU is unreliable for short text.
Can I use several references?
Yes, one per line. The best-scoring reference is used.
Will it match sacreBLEU?
Not exactly. Tokenization and smoothing differ.

Features

  • P1 to P4 shown
  • Brevity penalty
  • Multiple references

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Vectors | Essence of linear algebra

    3Blue1Brown

    Embeddings and vectors start with linear algebra intuition.

    Watch on YouTube
  • Finding mean, median, and mode

    Khan Academy

    Baseline stats for accuracy and evaluation helpers.

    Watch on YouTube