How We Actually Measure Whether an LLM's Output Is Good - BLEU, COMET and BLEURT

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product. An AI model writes a paragraph. It sounds fluent. It looks convincing. But how do you know whether it's actually good? This deceptively simple question has occupied researchers for more than two decades. Long before ChatGPT, machine translation researchers faced exactly the same problem. Human evaluation was expensive, inconsistent, and painfully slow. If every new model required thousands of humans to compare translations, research would crawl. That necessity gave rise to BLEU , one of the most influential evaluation metrics in AI history. Years later, as language models became better at paraphrasing and reasoning, BLEU started to show its age. Researchers responded with learned metrics like BLEURT and COMET , which use neural networks to judge language much more like humans do. Interestingly, this mirrors software engineering itself. We first wrote simple unit tests, then integration tests, and today we increasingly rely on sophisticated observability systems. Evaluation metrics for LLMs have undergone a similar evolution. Let's see why. Before BLEU: The Evaluation Bottleneck Imagine you're building Google Translate in 2001. Every time your team improves the model, someone has to read thousands of translated sentences and score them. Suppose a single sentence pair takes only 20 seconds to judge. Evaluating 50,000 sentences would require nearly 280 human-hours . Now imagine dozens of experiments every week. Evaluation—not training—quickly becomes the bottleneck. Researchers at IBM, led by Kishore Papineni , introduced BLEU (Bilingual Evaluation Understudy) in 2002 to automate this process. Their idea was surprisingly simple: If a machine translation resembles what professional translators write, it's probably good. This became one of the most cited papers