Practitioner Guide

LLM Evaluation in Production

The difference between teams that ship models weekly and teams that fear every change is a promotion gate. What to measure, how to automate judgment you can trust, and where evaluation data is allowed to live.

Eval layers a production stack actually needs

50–100

Golden examples to start a useful suite

Models promoted without passing the last model's evals

24/7

Online monitoring after the gate — drift never sleeps

The Stakes

Every Unevaluated Model Is a Production Incident on Layaway

LLM systems change constantly — prompts get edited, models get swapped, retrieval corpora grow, providers push silent updates. Each change can improve one behavior and quietly break three others, and unlike conventional software, nothing crashes when it happens. The output just gets worse, and the first detector is a customer. Evaluation is how you find out before they do: a measured baseline, a suite of checks every change must clear, and monitoring for what the suite couldn't foresee.

In the AI Scaling guide we make the case that evaluation gates are one of the four operational fixes separating the teams that deploy models daily from the majority that can't. This page is the practitioner's version of that fix: what the gate is made of, layer by layer.

Anti-Patterns

How LLM Evaluation Goes Wrong

Three patterns account for most quality incidents we see in the field. All three are cheaper to fix than the incidents they cause.

"It looks good" is the whole test plan

Someone pastes five prompts into a chat window, nods, and the model ships. Two weeks later a customer finds the failure mode nobody typed. Vibes-based QA doesn't scale past the demo it was invented for.

No baseline — 'better' has no number attached
No coverage — the hard cases are exactly the ones nobody tries
No memory — last month's regression comes back and nobody notices

Benchmarks stand in for your workload

MMLU and leaderboard scores measure a model's general ability — not whether it extracts your contract clauses or answers your support tickets. Public benchmarks are for shortlisting models, never for promoting them.

Benchmark tasks rarely resemble production tasks
Contamination inflates scores on public test sets
A 2-point leaderboard gap says nothing about your domain

Evaluation happens once, at selection

The team runs a bake-off, picks a winner, and never measures again. But prompts change, models get swapped, retrieval indexes drift, and providers update behavior on their schedule — quality is a moving target that needs standing instrumentation.

Prompt edits ship without regression runs
Provider model updates silently change outputs
RAG quality decays as the corpus and index drift

Architecture

The Five-Layer Evaluation Stack

Each layer catches what the others can't. Skipping one doesn't save work — it relocates the work to an incident channel.

Golden datasets

Regression

A versioned set of real inputs with known-good outputs — your unit tests for model behavior. Every prompt change, model swap, or retrieval tweak runs against it before anything ships. Start with 50–100 examples drawn from production, not imagination.

LLM-as-judge

Judgment

A strong model scores outputs against a rubric — correctness, completeness, tone — at a scale humans can't match. Judges need calibration against human labels and periodic audits; an unexamined judge is just automated vibes.

Human review

Ground truth

Domain experts label a sample of outputs to calibrate the judges and catch what rubrics miss. Expensive per item, so spend it where it counts: judge calibration, dispute resolution, and the failure modes that reach customers.

Retrieval & faithfulness

RAG-specific

RAG systems fail in two separable places: retrieval (did the right context surface?) and generation (did the answer stay faithful to it?). Measure both independently — most 'hallucination' complaints in production are retrieval misses wearing a disguise.

Production monitoring

Online

Sampled scoring of live traffic, user feedback signals, latency and cost per request on the same dashboard as quality. Offline evals gate what ships; online monitoring catches what the gate couldn't foresee.

Tooling exists at every layer — promptfoo and DeepEval for regression suites, Ragas for RAG metrics, Langfuse and Arize Phoenix for tracing and online scoring, and cloud-native options like Amazon Bedrock's evaluation jobs for teams running Claude and other foundation models on AWS. The stack is the easy part; the discipline below is what makes it bite.

The Discipline

Four Rules That Turn Evals Into a Gate

Metrics inform; gates decide. These are the rules that move evaluation from a dashboard nobody checks to a pipeline step nobody bypasses.

Version datasets like code

Golden sets live in git or a registry with versions, owners, and review. When an eval fails, you need to know whether the model changed or the test did — and un-versioned datasets can't answer that.

Gate promotion, don't advise it

Evals belong in the deployment pipeline with pass/fail authority: no model, prompt, or retrieval change reaches production without clearing the suite the last version cleared. Advisory dashboards get ignored the week a deadline hits.

Keep evaluation behind the boundary

Eval datasets are distilled from your most sensitive data — real contracts, real tickets, real records. Run evaluation where that data is allowed to live, under the same governance boundary as inference. Shipping your golden set to a third-party eval SaaS is a data transfer decision, not a tooling default.

Feed production back into the suite

Every incident, thumbs-down, and support escalation is a candidate golden example. The suite that catches next quarter's regressions is built from this quarter's failures — close the loop on a schedule.

The boundary rule deserves emphasis: evaluation datasets are distilled from exactly the data your governance policy exists to protect. Where evals run is a sovereignty decision — the full argument is in the Sovereign AI guide.

In Practice

Evaluation Is a Pipeline Stage, Not a Project

In our engagements, the evaluation harness ships with the platform, not after it. The six-week rapid strike timeline includes the promotion pipeline precisely because retrofitting a gate onto a live service is harder than building on one — the first model through the pipe establishes the baseline every successor must beat.

That's the shape of our AI consulting work generally: LLM evaluation and governance as a first-class capability alongside MLOps pipeline discipline — versioned golden sets, calibrated judges, gated promotion, and online monitoring, operated by US-based platform engineers on infrastructure you control.

FAQ

LLM Evaluation Questions, Answered

LLM evaluation is the practice of measuring whether a language model system — the model plus its prompts, retrieval, and tools — actually performs its task correctly, and continues to as things change. In production it spans offline regression testing against golden datasets, automated scoring with LLM-as-judge, human review for calibration, RAG-specific retrieval and faithfulness checks, and online monitoring of live traffic. The goal is a promotion gate: no change ships unless it measures at least as well as what it replaces.

Collect 50–100 real examples from your actual workload — production logs beat invented test cases — and record what a correct output looks like for each. Score a baseline run, then wire the suite into your deployment pipeline so every prompt or model change reruns it. Add an LLM-as-judge for qualities that string matching can't capture, and calibrate the judge against a sample of human labels before trusting it. Expand the suite from production failures as they occur.

LLM-as-judge uses a strong model to score another model's outputs against a rubric — enabling evaluation at a scale and cost human review can't match. It's trustworthy exactly to the degree it's calibrated: compare judge scores against human labels on a sample, measure agreement, and re-audit periodically. Known failure modes include position bias, verbosity bias, and self-preference when a model judges its own family. Calibrated judges are a production staple; uncalibrated ones are noise with confidence.

RAG adds a failure point before generation: retrieval. Evaluate it separately — did the retriever surface the passages needed to answer? — using metrics like recall and precision over a labeled set. Then evaluate generation for faithfulness: does the answer stay grounded in the retrieved context? Frameworks like Ragas formalize these dimensions. The separation matters diagnostically, because retrieval failures and generation failures have completely different fixes.

The open-source layer is mature: promptfoo and DeepEval for suite-style regression testing, Ragas for RAG metrics, and Langfuse or Arize Phoenix for tracing plus online evaluation. Cloud platforms bundle their own — AWS Bedrock ships model evaluation jobs for foundation models, including Anthropic's Claude family on the Bedrock platform. The tool matters less than the discipline: versioned datasets, a pipeline gate, and calibrated judges make any of these stacks work; without them, none do.

They make it more important and easier to govern at once. More important, because you own the promotion decision — no provider is silently improving the model behind an API for you. Easier to govern, because evaluation runs where the data lives: golden sets distilled from sensitive records never leave your perimeter, which keeps evals inside the same governance boundary as inference. For hybrid estates, the same suite scored against both your self-hosted models and external APIs is exactly the evidence that boundary policy needs.

AI Scaling Guide

Evaluation gates are one of four operational fixes that get teams deploying models daily — the full picture.

Sovereign AI Guide

The governance boundary that decides where models run and where eval data may travel.

Open WebUI on Kubernetes

The interface layer of the self-hosted stack this guide's promotion gates protect.

AI Consulting

Evaluation harnesses and governance are a core capability of our AI consulting practice.

MLOps Solutions

The pipeline discipline — CI/CD, GitOps, promotion — that evaluation gates plug into.

Case Study: Six-Week AI Rapid Strike

Production inference in six weeks — including the promotion pipeline models ship through.

Evaluation Is Where AI Programs Grow Up

There's a reliable maturity marker in enterprise AI, and it isn't model choice or GPU count — it's whether the team can answer "did this change make the system better?" with a number. Early-stage programs argue from anecdotes: a stakeholder's impressive demo, a customer's bad screenshot. Mature programs argue from eval runs. The transition usually happens after the first quality incident that a fifty-example golden set would have caught, which is why our advice is always to build the small suite now rather than the comprehensive one later. Fifty real examples with a pipeline gate beats five hundred planned examples in a backlog, every time.

The evaluation conversation is also inseparable from the platform conversation. Teams running on hyperscaler AI platforms — Claude on AWS Bedrock, models behind Azure or Google endpoints — inherit useful primitives: managed evaluation jobs, logging, guardrail hooks. Teams running open-weight models on their own Kubernetes inherit something different but arguably more valuable: total control over where evaluation data lives and total visibility into what changed when quality moved. Most enterprises end up hybrid, and the evaluation suite becomes the connective tissue — the same golden sets and rubrics scored against every backend, producing the comparable evidence that routing and governance decisions actually need.

What makes this discipline affordable is that it compounds. The first golden set is a week of annoying curation; every subsequent one is cheaper because the harness, the judge calibration, and the pipeline wiring already exist. Production failures stop being pure cost and start being test cases. Model upgrades stop being leaps of faith and start being afternoon experiments — run the suite, read the diff, promote or don't. That loop, more than any individual metric, is what evaluation buys: the ability to change AI systems at the speed the ecosystem changes, without gambling the customer experience on each move.

Ready to make AI operational?

Whether you're planning GPU infrastructure, stabilizing Kubernetes, or moving AI workloads into production — we'll assess where you are and what it takes to get there.

Book a readiness assessment Call Us Directly

US-based team · All US citizens · Continental United States only