Blog / observability
Tagged “observability”
What to collect when you’re evaluating an AI agent
Your eval score moved seven points and nobody can say why. What you have to record for a number about an AI agent to still mean something six weeks later.
Evals, part 1: when “it looks fine” stops working
GenAI output is probabilistic, so the old question, did it match the label does not work. Why AI products fail silently, and why spot-checking cannot catch it.
Evals, part 2: choosing the right ones for your product
A stack, seven product archetypes, and the one question that decides most of your eval design: is there a right answer to compare against?
Evals, part 3: what a year of building them taught us
Four principles, how to build an LLM-as-a-judge forwards rather than backwards, the five metrics to start with, and why robustness comes from stacking imperfect layers.