Evaluating LLM agents in production: a source-aware, calibrated approach

Most teams shipping LLM agents hit the same wall. The demo is great, users are happy, and then someone asks the question nobody has a good answer to: how do we know the next version didn’t get worse? The honest answer is usually “a few of us read some sample answers and it looked fine.” That doesn’t survive scale, it drifts with whoever is reading that day, and it gives you no defensible signal before a release.

This post walks through how to build an evaluation system that does survive scale. It’s written at the architecture level (the patterns, the frameworks, and the decisions that matter) rather than the line-by-line implementation. The stack here is Python, DeepEval (with its G-Eval metric), and a batch pipeline running on managed AWS infrastructure, but the ideas transfer to any equivalent tools.

---

The core problem: “good” isn’t a metric

A retrieval-augmented agent answers a question by pulling documents from some source and composing a response. When you evaluate that response, “is this good?” is the wrong question, because you’ll never get a consistent answer to it, from a human or a model.

There are really four things that can go wrong, and most pipelines flatten them into one “bad answer” bucket:

1. The answer is grounded in the evidence the agent retrieved.
2. The answer is fluent but pulled from the model’s own background knowledge, not the retrieved evidence.
3. The agent fell back to a generic web search when its primary retrieval came up empty.
4. Retrieval failed and the agent answered anyway.

Cases 2-4 all look confident. Telling them apart is the whole game, and you cannot do it after the fact from logs alone. You need the evidence the model actually saw, captured at the moment it saw it.

---

Architecture at a glance

The system splits cleanly into two phases: collection (run the agent, capture responses and evidence) and evaluation (score those responses against a rubric). They’re decoupled through storage, which is what lets the whole thing run unattended and lets each phase scale independently.

The key architectural decision is that evidence is a first-class output of collection, stored alongside the responses. Everything downstream depends on it.

---

Step 1: Capture the evidence, not just the answer

To evaluate grounding, you need the exact passages the agent retrieved, verbatim, at retrieval time. A reconstruction after the fact (“let me re-run the search and see what comes back”) is not the same thing; retrieval is non-deterministic and the index moves.

The pattern that works: instrument the agent’s tool-call stream. Modern agents emit a Server-Sent Events (SSE) stream as they work: tool calls, tool results, tokens. A collector listens to that stream and extracts, per response, the tool calls the agent made and the source material each tool returned. That evidence is written to a sidecar file (here, a JSON Lines file keyed by test ID) next to the response CSV.

Two things make this survive contact with a real pipeline:

• Upsert by test ID. Re-collecting a failed or empty response should replace that row and that evidence entry, and leave everything else untouched. Treat both the CSV and the sidecar as merge-by-key stores, not append-only logs. That makes retries cheap and idempotent.
• Classify provenance at capture time. While you have the stream, tag each response with which of the four modes above it fell into. You will never have better signal than right now.

---

Step 2: Score named dimensions, not vibes

Instead of one “quality” score, define an explicit rubric of named dimensions, each with written pass criteria and fail triggers. For a research agent these might be groundedness, citation correctness, retrieval relevance, completeness, recency, appropriate scope, and directness. The exact list is domain-specific. What matters is that each dimension is a separate, checkable question.

Each dimension is scored by an LLM judge using G-Eval, the DeepEval metric that turns a rubric into an evaluation. Rather than asking the judge for a free-form “7/10,” G-Eval runs a set of written evaluation steps and maps the result onto defined score bands, which makes scoring far more consistent than an open-ended number.

Crucially, the final verdict is a gate, not an average:

Averaging hides failures: a response can score well overall while being flatly wrong on the one dimension that matters (a fabricated citation, say). A gate says a prompt passes only if every dimension marked required for that question passes, and when it fails, it tells you which dimension failed. “Failed on citation correctness” is an actionable defect; “scored 0.62” is not.

A subtle but important detail: most of these dimensions are reference-free. There’s no single “correct answer” to diff against, because the underlying questions genuinely don’t have one. The fix isn’t a gold answer. It’s a gold rubric: specific, checkable criteria that turn a subjective judgment into a consistent one.

---

Step 3: Fail cheap before you fail expensive

LLM judge calls cost money and time. Before spending either, run deterministic pre-validation: cheap, code-only checks that catch obvious breakage, such as empty responses, malformed structure, missing required fields, error markers from collection. Anything that fails a deterministic check never reaches the judge.

This keeps runs fast and cheap, and it keeps your judge metrics clean, because you’re not asking a model to “evaluate” a row that’s obviously an error.

---

Step 4: Calibrate the judge before you trust it

Here’s the step most teams skip: an LLM judge can be confidently wrong, so you have to measure it before you gate a release on it. You don’t just deploy the judge; you prove it agrees with people.

The method is a calibration loop against a human-labeled gold set. Real evaluators score a sample of responses pass/fail per dimension. You then run the automated judge over the same sample and measure agreement, plus its false-positive rate (judge says pass, human says fail) and false-negative rate.

The disagreements are the gold. In one calibration pass, two patterns explained most of the gap: the judge scored holistically where humans checked claims one at a time, and it rewarded a response for gracefully admitting it couldn’t answer instead of penalizing it for not answering. Both were fixed by rewriting rubric steps and score anchors, not by touching model weights.

Set an explicit trust bar per dimension (for example, “a dimension isn’t allowed to gate unattended until the judge agrees with humans at least 85% of the time on it”). Track movement toward it. A real calibration cycle might move agreement from ~69% to ~77%, with false positives and false negatives both dropping several points: measurable, defensible progress, and a number you can put in front of a release board.

The important mindset shift: the gold set is a design tool, not a final exam. Label a small set first and let real disagreements shape the rubric, rather than treating labeling as validation you do at the end.

---

Step 5: Make it multi-target from the start

Once one product has a working evaluator, a second one shows up with different dimensions, different thresholds, and a structurally different response type. The wrong move is to fork the pipeline. The right move is to make the engine refuse to know anything domain-specific.

Everything that varies between products lives in external configuration (dimension definitions, per-dimension thresholds, pre-validation checks, metadata, and the judge’s framing prompt), loaded at runtime. The engine code stays identical.

This is what turns a pipeline into a platform: onboarding a new team becomes writing a config bundle, not editing the engine. It only works if you are disciplined about the boundary. The moment domain logic leaks into the engine, you’re back to forking.

A pinning note from experience: judge frameworks evolve, and a minor version bump can remove the exact hook you rely on for per-target prompt framing. Pin the framework version, and re-run your calibration set before ever bumping it. Your judge’s scores are only comparable over time if the thing producing them is stable.

---

Step 6: Run it unattended

None of this is useful if it only runs on someone’s laptop. The batch pipeline is fully automated on managed infrastructure, defined as infrastructure-as-code (Terraform):

A new batch of responses lands in object storage; a storage event fires a small function that triggers an orchestrated workflow (managed Airflow); the workflow runs the evaluation in a container; results and a regression diff against the prior run are written back. Every dimension is compared run-over-run, so a regression surfaces before release, not after a user finds it. Reports are generated for humans (interactive charts, PDF summaries) so the output is legible to people who won’t read a raw CSV.

---

Takeaways

If you’re building something similar, the transferable lessons:

• Capture evidence at collection time. Grounding can’t be evaluated from logs after the fact. Instrument the agent’s tool-call stream and store retrieved passages as a first-class artifact.
• Score named dimensions and gate, don’t average. A gate tells you what failed; an average hides it.
• Fail cheap first. Deterministic pre-checks before any LLM call keep runs fast, cheap, and clean.
• Calibrate against humans, and treat it as ongoing. A judge earns the right to gate a release by measurably agreeing with people. Set a trust bar and track it.
• Keep the engine domain-agnostic. Configuration, not forks, is what lets one evaluation engine serve many products.
• Automate it. An evaluation system that requires a human to run it isn’t a release gate. It’s a chore people will skip.

The tooling for LLM evaluation is increasingly a commodity. The durable value is in the parts that are specific to your system: the evidence you capture, the rubric you calibrate, and the discipline that keeps one engine serving many teams.