BLEU Score
Compute BLEU with brevity penalty against references.
About this tool
Enter a hypothesis (the machine output) and one or more reference sentences, one per line, and get a BLEU score with the four n-gram precisions and the brevity penalty. The text is lowercased and punctuation is removed before comparing.
How to use
How it works
BLEU is the geometric mean of the clipped (modified) n-gram precisions for n = 1 to 4, multiplied by a brevity penalty that punishes outputs shorter than the reference. A zero precision is floored at 1e-9 instead of giving exactly 0, which is why a short sentence with no matching 4-gram scores a tiny non-zero value. With several references it scores against each and keeps the highest.
BLEU = BP · exp( (1/4) Σₙ ln pₙ ) BP = min(1, e^(1 − r/c))
Worked example
Cat on the mat
- Input
- Hypothesis: the cat sat on the mat Reference: the cat is on the mat
- Output
- BLEU 0.003344; P1 0.833, P2 0.6, P3 0.25, P4 0; brevity penalty 1
- Walkthrough
- Five of six words match (P1 = 5/6), but no 4-gram matches, so P4 is 0 and drags the geometric mean down. An identical hypothesis and reference give 1.
Limitations
- Sentence-level BLEU is noisy; it is designed for corpus-level evaluation.
- No smoothing method other than the 1e-9 floor, so short sentences score near zero.
- Not the same implementation as sacreBLEU, so numbers will not match published results.
- Case and punctuation are ignored. Tokens are split on whitespace after cleaning.
Questions
How do I calculate a BLEU score?
Why is my BLEU so low for a short sentence?
Can I use several references?
Will it match sacreBLEU?
Features
- P1 to P4 shown
- Brevity penalty
- Multiple references
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Vectors | Essence of linear algebra
3Blue1Brown
Embeddings and vectors start with linear algebra intuition.
Watch on YouTubeFinding mean, median, and mode
Khan Academy
Baseline stats for accuracy and evaluation helpers.
Watch on YouTube
- DeepLearning.AI
DeepLearning.AI
Courses and explainers for modern AI tooling.
- Hugging Face docs
Hugging Face
Model hub docs next to tokenizer and embedding helpers.