Building a shared quality and release-gating platform for production AI

The Challenge

The client had already shipped an AI agent answering investment research questions from internal and third-party document sets. It worked, and people liked it, which was exactly the problem. Every new model version, every prompt change, got the same quality check: someone read a few dozen sample answers and said “yeah, that looks good.”

That process doesn’t survive scale. It drifts with whoever’s reading that day, and it has no answer for the question that matters before a release: how do you know the new version didn’t get worse? For a product answering finance questions with real confidence, a wrong answer delivered fluently isn’t a backlog item. It’s a credibility problem, sometimes a compliance one.

They needed a release gate. And because a second team inside the same organization had the identical problem on a different surface, they needed one engine both teams could adopt, not a one-off script wired to a single agent.

The Approach

1. Instrumented collection, not just logging. Evaluating grounding requires the evidence the model actually saw, not a reconstruction after the fact. Kayzn built a collection layer that captured, per response: the exact tool calls made, the retrieved source material verbatim at the moment of retrieval, and a provenance classification: was the answer genuinely grounded in retrieved evidence, pulled from the model’s own background knowledge, a fallback to generic web search, or a retrieval failure. Four distinct failure modes most pipelines flatten into one “bad answer” bucket.

2. A judge that scores named dimensions, not vibes. “Is this answer good?” isn’t a question you get a consistent answer to. So the rubric never asks it. Every response is scored across explicit dimensions (groundedness, citation correctness, retrieval relevance, completeness, recency, and others, fifteen in total across the two products’ rubrics), each with written pass criteria and fail triggers. An LLM judge scores each one against a structured rubric (G-Eval) with defined bands, rather than free-form 1-10. A prompt passes only if every dimension marked required for that specific question passes: a gate, not an average. Cheap deterministic checks run first to catch the obvious breakage before any LLM call is spent forming an opinion.

3. Calibrating the judge before trusting it. An LLM judge can be confidently wrong, so nothing about it was taken on faith. Kayzn built a calibration loop against a hand-labeled gold set (real evaluators scoring pass/fail per dimension) and measured how often the automated judge agreed. Kayzn traced disagreements to specific patterns: the judge scored holistically where humans checked claim-by-claim, and it rewarded a response for admitting failure instead of penalizing it for not answering. The rubric and score anchors were then revised against those observed gaps. Judge-to-human agreement increased from 69.2% to 76.9%, false positives fell from 9.9% to 7.7%, and false negatives fell from 20.9% to 15.4%, measurable progress toward the 85% per-dimension agreement bar Kayzn set as the point at which a dimension is trusted to gate unattended.

4. Turning the pipeline into a platform. A second AI product needed different dimensions, thresholds, and judge tuning: a different rubric for a structurally different response type. Rather than fork the pipeline, Kayzn redesigned the engine to be target-agnostic: dimension definitions, thresholds, pre-checks, metadata, and judge framing live in external configuration bundles instead of engine code. Two distinct AI products were onboarded this way, creating a repeatable path for additional teams to bring their own prompts, rubrics, thresholds, and metadata without rebuilding the engine.

5. Automated delivery and regression detection. New response batches land in cloud storage, trigger an orchestrated workflow, and run in containerized compute defined as infrastructure-as-code. The pipeline supports multiple model versions and environments, while each run is compared with prior results by dimension so regressions surface before release.

The Results

Quality

• Judge-to-human agreement increased from 69.2% to 76.9%, against an 85% per-dimension target set as the trust bar for unattended gating.
• False positives fell from 9.9% to 7.7%; false negatives fell from 20.9% to 15.4%.
• Failures became actionable. For example, “failed on citation correctness” instead of a low score with no explanation.

Delivery

• Automated batch collection, pre-validation, source-aware evaluation, reporting, and regression analysis replaced subjective manual spot checks.
• A four-way response-provenance taxonomy made grounded, model-knowledge, web-fallback, and retrieval-failure modes separately visible and trackable.

Scale

• One engine supports two distinct evaluation targets, multiple model versions, and production and pre-production environments.
• Configuration-driven onboarding gives additional teams a repeatable way to bring their own prompts, rubrics, thresholds, and metadata without changing the evaluation engine.

What Kayzn Would Do Differently

Calibration should have started before the rubric was “done,” not after. The human-labeled gold set was initially treated as final validation, but it proved more valuable as a design tool. Every rubric improvement came from an observed disagreement pattern rather than theorizing about what the judge should catch. On a future engagement, Kayzn would label a small gold set first and let real disagreements shape the rubric from day one.

Key Learnings

A shared platform only works if the engine refuses to know anything domain-specific. Keeping dimensions, thresholds, and scoring rules in configuration is what allowed a second product to adopt the platform without a fork and created a clean path for future teams.

An evaluation system has to earn trust, not assert it. Building the judge is the easy part. Proving, with real disagreement rates against real human labels, that its opinion is worth gating a release on is the actual work, and it never really finishes.

Reference-free doesn’t mean unaccountable. Most of these dimensions have no single “correct answer” to check against, because the underlying questions genuinely don’t have one. The fix isn’t a gold answer, it’s a gold rubric: specific, checkable criteria that turn a subjective judgment into something consistent, explainable, and portable across teams.

---

Client and internal system names withheld per engagement confidentiality. Technical details, architecture, and results are described as delivered.

This is the kind of problem Kayzn takes on: turning a product-specific quality problem into dependable infrastructure that can serve multiple products without multiplying maintenance.

Need to turn AI evaluation from manual review into a dependable release process? Kayzn can design and deliver the collection, calibration, evaluation, and platform architecture. Book a Process Fit Call →