Free, and no data changes hands

Send us the one criterion you trust least.

The agent can be live, in a pilot, being built, or still an idea. Not your traces. Just the rule, in whatever words you already use for it. We write back on what it would take to make it decide something, on one page.

What comes back

  • Whether it can be scored at all, said plainly. Some criteria as written cannot be, and knowing which of yours are in that category is worth more than a tidier version of the ones that are.
  • How we would score it, with the pass criteria and the fail triggers written out, in the form a judge could actually run.
  • What a human would have to label before an automated judge on it could be trusted, and roughly how many, so you know the cost in your expert's time rather than in ours.
  • Where it would go wrong. Which cases sit on the boundary, and what a judge would probably get wrong about them at first.

What a criterion is, if that is not a word you use

One rule you would use to decide whether an answer from your AI agent is good enough. Not the whole definition of a good answer, one rule inside it.

  • Every claim has to be supported by a document we actually retrieved.
  • It must not answer at all if the policy is out of date.
  • The citation has to point at the paragraph the number came from.

Pick the one you would least want to bet a release on. The ones you are confident about are not where the interesting answer is. And if it is confidential, describe the shape of it rather than sending the real one: the answer is about how a rule like that gets scored, and that transfers.

Send it over

Six questions. Three about the AI agent, three about where to send the answer, and only one of them takes any thought.

01 / The AI agent

One line is plenty. "Answers policy questions for support staff."

Where is it today?

In whatever words you already use for it. If you would rather describe the failure than write the rule, do that instead: working out what the criterion should be is part of what we will send back.

02 / Where to send the answer

If there is something public, this helps us read it before we answer rather than replying in general terms.

We write back within two business days, even if it is to tell you yours is in the queue. No traces, no customer data and no NDA needed for this. We will use this to write back about your criterion, and to follow up once about whether we can help. Nothing goes on a mailing list unless you ask, and nothing is shared outside Kayzn. What we store.

The honest limits

This is a written answer about one rule, not an evaluation. It cannot tell you how often your AI agent is right, because that needs traces and labels. What it can tell you is whether the thing you are calling a criterion is the kind of thing a machine could ever check, which is the question most teams have not actually asked before building a suite on top of it.

We read these ourselves, so we take a small number at a time. If yours has to wait we will say so when we reply rather than leaving you wondering.

Would rather talk it through? Book an Evaluation Fit Call. Also free, and it ends in the same recommendation.