Blog / calibration
Tagged “calibration”
Evals, part 1: when “it looks fine” stops working
GenAI output is probabilistic, so the old question, did it match the label does not work. Why AI products fail silently, and why spot-checking cannot catch it.
Evals, part 2: choosing the right ones for your product
A stack, seven product archetypes, and the one question that decides most of your eval design: is there a right answer to compare against?
Evals, part 3: what a year of building them taught us
Four principles, how to build an LLM-as-a-judge forwards rather than backwards, the five metrics to start with, and why robustness comes from stacking imperfect layers.
Evaluating LLM agents in production: a source-aware, calibrated approach
How to move from subjective spot checks to a repeatable, evidence-based evaluation platform that can gate releases, and serve more than one product.