The Problem With Benchmarks
When organisations evaluate large language models for production use, the instinct is to reach for the published leaderboards. MMLU, HellaSwag, HumanEval — these benchmark scores are prominently displayed on every model card, and a model that tops a leaderboard naturally feels like a safe choice. The problem is that benchmark performance and real-world task performance are not the same thing, and conflating the two can send an engineering team months down the wrong path.
MMLU — the Massive Multitask Language Understanding benchmark — is a multiple-choice question suite designed to probe academic knowledge across dozens of domains. It is a reasonable proxy for general reasoning ability, but it tells you almost nothing about whether a model will draft persuasive email copy, extract structured data from messy invoices, or follow the particular instruction style your users expect. A model that scores two percentage points higher on MMLU may still produce worse outputs for your specific use case than the model it nominally beat. Benchmark contamination, where training data overlaps with evaluation sets, makes the situation worse: a model can appear capable on paper while having effectively memorised the test.
The conclusion is not that benchmarks are useless — they help narrow the field — but that they are a starting point, not a finishing line. Real evaluation has to be grounded in the actual work the model will be asked to do.
Building a Task-Specific Evaluation Suite
The most defensible evaluation approach begins with real inputs. Rather than constructing synthetic test cases from first principles, the better practice is to collect examples that represent genuine production traffic — the kinds of prompts, documents, and queries that users actually send. This immediately surfaces quirks that synthetic data misses: unusual phrasing, domain-specific jargon, edge cases that only appear after weeks of real usage.
Once a corpus of real inputs exists, the next step is to label expected outputs or, at minimum, to define what a good output looks like for each input type. This is slower and more expensive than running a benchmark, but it is the only way to build a ground truth that is actually aligned with your product's goals. The size of this evaluation set matters less than its representativeness. A few hundred well-chosen, carefully labelled examples will outperform thousands of hastily assembled ones when it comes to making reliable decisions about model quality.
It is also worth segmenting the evaluation suite by task type. If a model handles both summarisation and code generation, lumping those tasks together into a single aggregate score will obscure failures in either direction. Separate slices allow separate accountability, and they make it much easier to diagnose regressions when they occur.
Choosing Metrics That Suit Open-Ended Generation
Evaluation becomes significantly harder when model outputs are open-ended — when there is no single correct answer to compare against. Traditional string-matching metrics like BLEU and ROUGE were designed for translation and summarisation tasks where the space of acceptable outputs is relatively constrained. Applied to open-ended generation, they frequently mislead. A response that is fluent, accurate, and genuinely useful can score poorly on ROUGE simply because it paraphrases an expected answer rather than reproducing its exact wording.
Several approaches have emerged to fill this gap. Human evaluation remains the gold standard, but it is expensive and difficult to scale. Model-based evaluation — using a capable LLM as a judge — has become increasingly common as an intermediate option. A judge model can assess outputs along dimensions like factual accuracy, coherence, instruction following, and tone in a way that correlates reasonably well with human judgement, at a fraction of the cost. The risk is that judge models have their own biases and blind spots, which means their assessments should be periodically calibrated against human ratings rather than trusted unconditionally.
For tasks that do have a verifiable correct answer — data extraction, code that either runs or does not, classification with a known label set — execution-based or exact-match metrics are far more reliable than any text similarity measure. Where possible, designing tasks so that correctness is programmatically verifiable is a significant advantage, because it removes the subjectivity problem entirely.
Regression Testing for Prompt Changes
Prompt engineering is iterative by nature. Teams tweak wording, adjust system instructions, add few-shot examples, and restructure output formats in pursuit of better results. What is easy to miss in this process is that a change intended to improve performance on one class of inputs can quietly degrade performance on another. Without a systematic regression testing framework, these regressions often go undetected until they cause visible failures in production.
The solution is to treat prompts with the same discipline applied to code. Every meaningful prompt change should be run against the full evaluation suite before deployment, with results compared against the previous version. The comparison does not need to be exhaustive — a statistical summary of pass rates by task segment, combined with manual review of any cases that flipped from pass to fail, is usually sufficient to catch significant regressions. The key cultural shift is recognising that a prompt is not a configuration detail but a core part of the system, with the same potential to introduce bugs as any other code change.
Version controlling prompts, alongside evaluation results, also creates an audit trail that becomes valuable over time. When a new model version is released and the team wants to understand whether switching providers is worth the cost, historical eval runs make that comparison tractable rather than anecdotal.
Latency and Cost Are Part of Quality
Quality-focused evaluation suites tend to focus exclusively on output correctness, but in a production system, a response that is technically accurate yet takes twelve seconds to generate may be worse than a slightly less polished response that arrives in two seconds. User experience is shaped by the whole interaction, and latency is a first-class component of that experience.
Cost operates similarly. A model that produces marginally better outputs but at five times the inference cost may not represent a genuine improvement once the economics are factored in. Evaluating latency and cost per task, not just aggregate throughput, allows teams to make principled trade-offs. A use case that is latency-sensitive and cost-sensitive will have very different model requirements than a batch processing pipeline where neither constraint is binding.
The practical recommendation is to incorporate latency percentiles — particularly p95 and p99, which capture tail behaviour that averages hide — and cost per successful completion into the same evaluation dashboard as quality metrics. When all three dimensions are visible together, the team can reason clearly about what they are actually trading off when they consider switching models or changing providers.
Evaluation as an Ongoing Practice
Perhaps the most important framing shift is to treat LLM evaluation not as a one-time gate before launch but as a continuous engineering discipline. Models change, prompts evolve, user behaviour shifts, and the distribution of production inputs drifts over time. An evaluation suite that was well-calibrated at launch will gradually become less representative unless it is actively maintained.
Teams that build this discipline early — collecting real examples, defining clear metrics, running regressions on every change, and tracking quality alongside latency and cost — are in a fundamentally better position than those who rely on leaderboard scores and intuition. Benchmark numbers will always be useful for initial shortlisting, but the organisations that build reliable AI products are the ones that do the harder, slower work of measuring what actually matters to their users.