11 LLM Evaluation Methods: From BLEU to Safety Scoring

LLM evaluation is not a solved problem. The article opens with a concrete failure: a model can produce the correct answer yet score near zero on BLEU if it paraphrases instead of matching the reference wording verbatim. This fragmentation exists because different applications demand different correctness assumptions. Translation needs precise wording (BLEU); summarization tolerates rewording as long as key content is covered (ROUGE); subjective tasks like tone or instruction-following have no reference answer at all. The guide maps this landscape by grouping metrics into families: n-gram overlap (BLEU, ROUGE), semantic similarity (BERTScore), model-as-judge approaches (G-Eval, LLM-as-Judge, juries), human annotation, and task-specific methods (DAGs for deterministic rules, trajectory accuracy for agent behavior, multi-turn eval for conversation consistency). A key pattern emerges: production systems interleave multiple methods rather than picking one. BLEU alone is unsafe; pairing it with BERTScore (which handles paraphrase) or human calibration validates the choice. Similarly, a single LLM judge carries predictable bias, but several judges from different model families average away noise without amplifying their correlated errors. Safety evaluation sits apart—it is a gate, not a quality metric, because one leaked PII instance invalidates a high average score.