Blog
Notes from the work.
Practical lessons from real projects: how something was built, what worked, what did not, and what we would do differently next time. No pitch, and no login. Take what is useful.
We put four AI judges on trial. Here is who got it right, and what it cost.
Before you let a model grade your AI agent, someone has to grade the model. We ran four judges, including Jev, TypeSafe’s new System One model, over 230 support conversations with a hidden answer key.
What to collect when you’re evaluating an AI agent
Your eval score moved seven points and nobody can say why. What you have to record for a number about an AI agent to still mean something six weeks later.
Evals, part 1: when “it looks fine” stops working
GenAI output is probabilistic, so the old question, did it match the label does not work. Why AI products fail silently, and why spot-checking cannot catch it.
Evals, part 2: choosing the right ones for your product
A stack, seven product archetypes, and the one question that decides most of your eval design: is there a right answer to compare against?
Evals, part 3: what a year of building them taught us
Four principles, how to build an LLM-as-a-judge forwards rather than backwards, the five metrics to start with, and why robustness comes from stacking imperfect layers.
Evaluating LLM agents in production: a source-aware, calibrated approach
How to move from subjective spot checks to a repeatable, evidence-based evaluation platform that can gate releases, and serve more than one product.