Skip to main content
ZICQ

Wiki Concepts

BLEU / ROUGE

Concepts
Aliases: BLEU ROUGE text generation metrics ·2026-09-14

BLEU / ROUGE

BLEU and ROUGE are the most classic automatic metrics for NLP text generation tasks. Still have reference value in the LLM era, but must be paired with human evaluation.

BLEU (Bilingual Evaluation Understudy)

  • Purpose: evaluate machine translation quality, also used for other generation tasks.
  • Principle: calculate n-gram overlap (1-4 grams) between generated and reference text, multiplied by brevity penalty (to discourage short answers).
  • Range: 0-1 (or 0-100), higher is better.
  • Characteristic: emphasizes precision, unfriendly to paraphrases.
from sacrebleu import sentence_bleu
score = sentence_bleu(hypothesis, [reference])

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

  • Purpose: evaluate summarization quality (also used for other generation tasks).
  • Variants:
    • ROUGE-N: n-gram recall (complementary to BLEU)
    • ROUGE-L: longest common subsequence (LCS), considers sentence-level structure
    • ROUGE-W: weighted LCS, rewards consecutive matches
    • ROUGE-S: skip bigram
  • Range: 0-1, higher is better.
  • Characteristic: emphasizes recall, unfriendly to missed content.
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(['rouge1','rouge2','rougeL'], use_stemmer=True)
scores = scorer.score(reference, hypothesis)

Problems in the LLM era

  • Paraphrase blindness: "car" vs reference "vehicle" scores 0.
  • Length sensitive: long texts artificially low BLEU/ROUGE.
  • Quality not reflected: fluent but factually wrong sentences can still score high.
  • Factuality not assessed: hallucinations can't be detected by BLEU/ROUGE.
  • LLM-as-a-Judge: use GPT-4 / Claude as judge (MT-Bench, AlpacaEval).
    • Pros: evaluate quality, fluency, factuality.
    • Cons: expensive, biased (judging itself).
  • Human Eval: human annotation (Chatbot Arena, HHEM).
    • Pros: gold standard.
    • Cons: expensive, slow, not reproducible.
  • Task-specific metrics: e.g., FactCC for summarization, EM/F1 for QA, pass@k for code.

Practical advice

  • Quick iteration with BLEU/ROUGE: cheap, reproducible, catches obvious regressions.
  • Final quality with LLM-as-a-Judge + human: complement BLEU/ROUGE blind spots.
  • Combine multiple metrics: relying on one metric is adversarial-friendly.