Evals, part 3: what a year of building them taught us

Part three of three, on measuring the quality of AI products.

Four principles

Part one covered why evals exist and part two covered how to choose them. This is what I actually learned doing the work: the stuff I would tell you over coffee before you write a single evaluator. Four principles.

1. Stay paranoid

Assume the model is quietly getting worse. Silent degradation is the default state of an AI product, not a rare failure.

Two things drift out from under you with no code change on your side. First, inputs change. The way people used AI a year ago isn’t how they use it now. They’ve stopped typing keyword-search fragments and started writing paragraphs of context. Your traffic distribution moves. Second, hosted models change. The provider ships a new version overnight and your product can degrade by morning while your diff is empty.

And here’s the trap: a green dashboard is a claim, not a fact. Aggregate metrics are exactly the thing that masks drift. The reported line stays flat and pretty while the real quality line slides underneath it. The line that matters is usually the one no dashboard is plotting.

So: audit the judge. Your LLM-as-a-judge is a model too, subject to the same drift. Periodically re-run judge-vs-human agreement so you know your measuring stick hasn’t bent.

2. Define the hill

The eval is the spec. Don’t hand your team a PRD and hope; hand them an eval. It cuts the translation layer between product and modeling, because it says precisely what “good” means in a form the team can run.

Then decompose “good” into atomic checks: small, independently measurable criteria that ladder up to the summit. Prefer many narrow evals over one broad one. Ten high-confidence checks beat a single check that tries to do everything. A good test of a check is reproducibility: run the same eval 100 times and it should score consistently. If it doesn’t, it’s too broad or too vague to trust.

And refresh your evals, or Goodhart’s law kills them. “When a measure becomes a target, it ceases to be a good measure.” The moment your team starts hill-climbing a proxy, they’ll game it, and the eval saturates. Retire and renew. The mental model I like: lots of small hills your team can actually climb, little goals along the way to the summit, not one lumpy plateau nobody can get up.

3. Respect the tails

Nobody experiences your average. Average scores are comforting and almost useless, because real harm lives in the rare, worst-case behavior that an average smooths right over.

So triage by severity, not frequency. One rare, fatal answer outweighs a pile of minor slips. Set thresholds by context: some metrics get a lenient bar, and on critical paths you want zero tolerance for high-severity failures. Frequent and important are not the same axis, and the tall bars on your failure chart are rarely the ones that will hurt you.

The practical move is to build the ugly dataset on purpose. Stratify your golden set into buckets: a production sample, the long tail, adversarial edge cases, and replays of failures that already shipped. Our team is building exactly this for our agents, and we’re putting it in a shared library so any team building AI products can pull from it instead of reinventing the nasty cases.

4. Read the transcripts

This is the most important eval activity, full stop. Evals aren’t QA you bolt on at the end. Reading your product’s actual output is what tells you which evals are even worth writing in the first place.

There’s a method for this, borrowed from grounded theory in the social sciences (Glaser and Strauss, 1967):

• Open coding. A domain expert reads real traces and writes open-ended notes, journaling the first failure they see in each one, with no fixed taxonomy. Just look and describe.
• Axial coding. Group those notes into a failure taxonomy and count each category. This is the step that sets your priorities.
• Iterate to saturation. Keep going until roughly 20 fresh traces reveal no new category (plan to review 100+).

The codes emerge from the data, then group into a taxonomy. You don’t start with the categories, you discover them. When you do this on a real agent, the priorities jump out. On one agentic task breakdown, the failure modes came out as something like 45% incorrect implementation, 40% incorrect plan, 15% wrong data selection, and now you know exactly what to fix first, and what to write a judge for.

If you only take one thing from these four: stay suspicious of your own dashboards, and go read the transcripts. Everything else here is downstream of those two habits.

LLM-as-a-judge: a PRD that runs forever

When you have too many acceptable answers for a human to check by hand, you reach for an LLM-as-a-judge (LLMJ): a model that scores your product’s output against a rubric you write. It’s fast, cheap, and scales. It’s also the piece people most often build backwards. Here’s how to build it forwards.

Start from the transcripts, not from a trendy metric

Don’t open with “let’s measure hallucination rate.” Start with real user interactions and let the failures tell you what to measure. The build has five phases:

1. Discovery. Pull ~100 real conversations. Have a domain expert, often the PM, read them and write quick notes on what went wrong. LLMs lack business context; only a human catches that “free shipping minimum” or “check-in policy” nuance.
2. Pattern recognition. Group similar failure notes into categories (“wrong delivery timeframe,” “shipping-policy error”). An LLM can help with the initial grouping, but a human has to confirm the categories make business sense.
3. Prioritize by counting. Count how often each failure type occurs. That count is your roadmap. It tells you what to fix, and what to judge, first.
4. Build the judge. Write a binary “LLM-as-a-judge” for your top failures. Did the interaction meet the quality bar or not? Make it binary. A pass/fail forces clarity and turns a fuzzy standard into an executable one.
5. Validate before trusting. Test the judge against human labels. Don’t rely on simple accuracy; look at true-positive and true-negative rate so you know exactly how your judge makes mistakes.

That whole sequence is the grounded-theory workflow from the principles above (open coding, axial coding, saturation) pointed at a specific goal: a judge you can run continuously.

The judge prompt is a PRD that runs forever

Here’s the reframe I want to land. Your judge prompt is a rubric, and that rubric is your PRD, except instead of sitting in a doc, it runs on every response, forever.

The Michelin Guide is the analogy I keep coming back to. You don’t control what a restaurant does, but you own the inspection process, and it comes down to five criteria: quality of ingredients, mastery of technique, harmony of flavors, personality of the cuisine, and consistency. Those are deeply subjective, and Michelin still turns them into a repeatable rubric that anonymous inspectors apply, year after year, in a cycle of re-evaluation.

That’s your job with a judge. You write the rubric; the AI applies it at scale on your behalf. Which means the quality of your product is only ever as good as the rubric you wrote.

Taste is our responsibility, don’t delegate it

This is the part I feel strongest about. Too often the rubric gets handed to the ML team or “somebody else,” and whoever writes it is, quietly, defining how the product behaves. That’s a product decision wearing an engineering costume.

Taste is literally our job. Writing the rubric should sit in the PM domain. If you decompose “good” into clear criteria and write the judge prompt yourself, you’ve done maybe 90% of the thinking before you ever walk into the room with your ML team, and now that conversation is informed instead of hand-wavy.

Three things that will bite you

Calibration is ongoing, not one-time. A judge outputs a score; you have to keep checking that score against humans. When the judge says “pass,” does a human agree on the same response? That agreement rate is your confidence in the judge. It’s garden maintenance: periodic, never finished.

When humans disagree, you need a tiebreaker. Annotators won’t always agree. That’s normal, not a crisis. Options that work: send the same task to multiple annotators and take the majority vote; when votes tie, escalate to an SME with deep knowledge of the relevant policy; and set clear rubrics up front so grading stays as objective as a subjective task allows. Our default tiebreaker is simple: the PM who owns the AI product is the final say. If the humans deadlock, they call it.

Judging is expensive. Running a judge means burning tokens on every trace, and it can cost roughly 10x more to evaluate a product than you’d expect. We’re tackling this by making the judge model configurable: swapping an expensive commercial model for an open-source model hosted in AWS, and using our existing human annotations to confirm the cheaper model stays aligned with the judgments we trust. Same rubric, same alignment check, a fraction of the cost.

The one-liner to remember: an LLM judge prompt is a PRD that runs forever, so write it like the product decision it is, and keep checking that it still agrees with a human.

The five metrics that matter

There are dozens of metrics you could track. Start with five. Layer the fancy, data-science-heavy ones on later, once these are solid. Each one maps to a question a traveler is actually asking, because that keeps the metric honest.

Metric	The traveler’s question	What it checks
Relevance	Is the AI answering my question?	The response addresses the intent, not the topic
Factuality	Can I trust this?	The answer is correct in the world, verifiable on the web
Hallucinations	Can I trust this?	Nothing fabricated: no invented facts, names, quotes or citations
Completeness	Is the answer complete and precise?	Every part of the question got answered
Groundedness	Is the answer complete and precise?	The answer reflects the source material it was handed

The difference between them is easier to see in worked examples than in definitions.

Metric	Given	A good answer	A bad one
Relevance	“Weather in Montreal tomorrow?”	“Sunny, around 5°C.”	“Montreal is a lovely city in Quebec…” True, and useless.
Factuality	“Who won Best Picture in 2024?”	“Oppenheimer.”	“Barbie.”
Hallucinations	An answer that cites a source	A citation you can go and check	“The New York Times called it a ‘revolutionary breakthrough’ in a 2024 article by David Chen,” where neither the article nor the journalist exists
Completeness	“Compare pros and cons of Bahamas vs Cancún in December.”	Pros and cons for both	Only Bahamas, in detail
Groundedness	A property’s cancellation rule, to summarize	“You have 30 days before the trip start date to cancel.”	“You have 45 days from booking date to cancel.”

Two of these deserve a note.

Hallucinations are caught from two directions, depending on where the truth lives. If the claim should have come from your own data, check groundedness against it. If it is a claim about the world, check factuality against a web search.

Completeness is the one that surprised us. We started spotting it in the comparison assistant: a user picks five properties, asks a question, and the answer only addresses three of them. And here’s the honest part: some of that is our own fault. We constrain the AI (“answer everything in X characters, in this format”), so it silently decides what to prioritize and what to drop. That’s a real product tension. Do we want to be exhaustive, or is it acceptable to miss a little? Either way, you can’t manage it if you’re not measuring it.

How we run evals

Knowing the metrics is one thing; running them repeatably across many teams is another. Here’s the shape of what we built so nobody has to start from scratch. The pattern generalizes even if the specific tools don’t.

Start with an observability platform. Everything downstream depends on capturing your product’s traces: the full record of each interaction, including inputs, tool calls, intermediate steps, and outputs, with metadata attached. We use LangSmith. The category is “LLM observability and tracing,” and there are several options (LangSmith, Langfuse, Arize Phoenix, Braintrust, and others). Think of it as the Splunk of your probabilistic product: it stores the traces, manages your datasets, and gives your evals something to run against. The single highest-leverage first move is to instrument your product so every interaction is captured. If you can’t see it, you can’t evaluate it.

Wrap it in an eval toolkit. We built a thin layer of utilities and adaptors around the observability platform. It runs next to your agent (or your ML model), fetches the traces, picks an evaluator, calls the model to score, and writes the score and reasoning back where you can see them. It plugs into CI/CD so you can gate a deploy on eval results, and the judge model is configurable, so you can point it at a cheaper hosted model to control cost. You don’t need our exact toolkit; you need some glue between your traces, your evaluators, and your pipeline.

Build a shared evaluator library. This is the piece I’d push hardest on for any org running more than one AI product. We kept watching teams rebuild the same evaluators, faithfulness and groundedness, and each one is genuinely costly to build and calibrate. So we made them vetted, reusable, and pluggable. Groundedness and faithfulness are the obvious candidates for a common “everyone passes this vanilla benchmark before shipping” bar, with teams free to add stricter, feature-specific checks on top. Centralizing the common evaluators turns a per-team tax into a one-time investment.

Give people a fast on-ramp. I built a Claude Code skill that takes a PRD and helps you produce an eval strategy, generate synthetic data, and write a first judge prompt. Building one product’s strategy this way, the skill surfaced a metric we didn’t have (Mean Opinion Score, for how natural synthesized speech sounds) and prompted scenarios I hadn’t thought of, like regional accents and interruptions mid-response. The synthetic data comes out as a CSV of persona, question and expected output. The result is a package (PRD, eval strategy, dataset) you can hand your team and say “can we run this and see if it works?” The lesson that transfers: lower the activation energy, or evals stay perpetually on next quarter’s list.

Worth saying: this isn’t only for LLM products. The same machinery works for classic ML, search-result relevance or a next-best-action recommender, anywhere you have non-deterministic output and want to know if quality degraded. If your traces are captured, you can evaluate them.

Stack your defenses: the Swiss cheese model

One last mental model, and it’s the one I’d want you to walk away with if you remember nothing else about how the pieces fit together.

The Swiss cheese model comes from James Reason’s work on accident causation in aviation and healthcare, the fields where a single missed check can kill someone, so they’ve thought harder than anyone about layered defense. The idea: picture your safety measures as slices of Swiss cheese stacked front to back. Every slice has holes. No barrier is perfect. But the holes are in different places on each slice, so a hazard that slips through the first hole gets stopped by the solid part of the next one. An accident only happens when the holes on every slice happen to line up and a straight path opens all the way through. Stack enough well-placed slices and that alignment becomes vanishingly rare.

Evals are exactly this, and the reason the analogy earns its keep is that it reframes the goal. People want to find the one perfect eval, the metric that catches everything. There isn’t one, and chasing it is a trap. The goal isn’t a flawless slice; it’s a stack where the holes don’t overlap.

So what are the slices? Each is a different kind of eval, and each has a characteristic blind spot:

• Offline benchmarks and golden datasets catch regressions on cases you already understand, but they’re blind to anything you didn’t think to put in the set. The map is not the territory.
• Red-teaming and adversarial testing catch the malicious and the weird (the injection attack, the edge case), but they don’t tell you how the product does on ordinary Tuesday-afternoon traffic.
• Automated evals and LLM-as-a-judge catch quality issues at scale, cheaply, on every response, but they inherit the judge’s own blind spots and drift, and they only measure what the rubric thought to ask about.
• Human review catches the nuance a rubric misses (the answer that’s technically correct and subtly off-brand, the business-context error no model would flag), but it’s slow and covers only a tiny sample.
• Production monitoring catches what real users actually hit, including drift and distribution shift no test anticipated, but it catches it after the answer already went out, not before.

Read that list and the point becomes obvious: every single layer, on its own, is porous. Ship on offline evals alone and production traffic walks right through the gaps in your dataset. Rely on the judge alone and it drifts without a human to catch it. Trust human review alone and you’re spot-checking 1% while 99% goes unseen.

The subtle part, the part teams get wrong, is independence. Two slices only cover each other’s holes if their holes are in different places. If your offline dataset and your LLM judge were both built from the same PM’s mental model of the product, they share the same blind spot, and stacking them buys you far less than it looks like. You’ve got two slices with holes in the same spot. Diversity of method (code checks, model judges, human eyes, live monitoring) is what staggers the holes. Correlated evals give you a false sense of depth.

Every eval has holes. Robustness doesn’t come from one perfect layer. It comes from stacking enough diverse, independent slices that the holes rarely line up. That’s the whole game: not a wall, but a stack.

Getting started

You don’t need all of this on day one. A realistic path:

1. Read your transcripts and code the failures.
2. Pick your archetype and decide reference-based vs reference-free.
3. Build a small offline golden dataset, including an ugly, adversarial slice.
4. Stand up the five metrics above as your baseline.
5. Turn your top failure into a binary LLM-as-a-judge and calibrate it against a human.
6. Instrument your product with an observability platform so every interaction is captured, and reuse shared evaluators instead of rebuilding groundedness and faithfulness from scratch every time.

None of this requires a big platform team to start. The first three steps you can do with a spreadsheet, a notebook, and an afternoon reading transcripts. The tooling matters later, once you’ve proven the habit is worth automating.

One thing to remember

Stay in the constant state of looking at your product’s output and asking: should I add another eval? That habit, not any single dashboard, is what keeps quality from sliding out from under you.

You gotta catch ‘em all.