Your AI Feature Needs an Evaluation Set More Than a Better Prompt

Here is a pattern we see constantly. A team ships an AI feature. It works. Two months later someone tweaks a prompt to fix a complaint, and three other things quietly get worse. Nobody notices until a customer does.

The team's instinct is to be more careful with prompts. The actual fix is to stop treating quality as an impression and start treating it as a number.

Why AI features regress silently

Traditional software fails loudly. A bad deploy throws exceptions, error rates spike, someone gets paged.

An AI feature that degrades produces perfectly well-formed responses that are simply worse. No exception, no alert, no spike on any dashboard you already have. The only signal is user frustration, which arrives weeks late and heavily filtered.

Worse, the causes are invisible from your side: a provider updates a model, your input distribution shifts as you win a different type of customer, a retrieval index goes stale. None of those are code changes, so none of them show up in a diff.

What an evaluation set actually is

It's a list of realistic inputs, each paired with a way to check the output. That's it. No platform required to start: a JSON file and a test runner will do.

The pieces that matter:

Real inputs. Pull them from production traffic, not from your imagination. Developer-invented test cases are systematically cleaner, shorter, and more polite than what users actually send.

Coverage of the distribution. Include the common cases, the long tail, and, most importantly, every case that has already gone wrong. A regression that has bitten you once should be impossible to reintroduce.

A grader per case. Not a single global metric. Different cases check different things.

Prefer deterministic graders

The instinct is to reach immediately for a model-as-judge. Resist it for as long as you can, because a surprising share of quality is mechanically checkable:

• Does the output parse against the expected schema?
• For extraction, do the field values exactly match the labelled answer?
• Is every citation a document that was actually retrieved?
• Does the answer contain a phone number when it must not?
• Did it stay under the latency and cost budget?

Deterministic graders are fast, free, and never drift. Build these first. For many features (extraction, classification, routing, structured generation) they cover almost everything you care about.

Use a judge carefully, and check it

For open-ended output, a model judge is the only practical option. Two rules make it trustworthy.

First, give it a rubric, not a vibe. "Rate 1–5 for helpfulness" produces noise. "Return fail if any factual claim is not supported by the provided context, otherwise pass" produces a signal you can act on. Narrow, binary, grounded.

Second, validate the judge against humans. Label fifty cases yourself, run the judge on them, and measure agreement. If it agrees 70% of the time, your metric has more noise than the effects you are trying to detect, and you will make decisions based on it that are worse than random.

Put it in CI, or it won't happen

An evaluation suite that runs when someone remembers is an evaluation suite that runs before launch and never again.

Wire it to pull requests. Any change to a prompt, a model version, a retrieval parameter, or a tool schema runs the suite and posts the deltas. Set a threshold that blocks the merge. Print the specific cases that changed verdict, because "accuracy dropped 2%" is not actionable but "these four cases now fail" is.

The cost is real but small: a few dollars and a few minutes per run for a suite of a hundred cases with a cheap judge. Compare that to shipping a regression.

Grow the set from failures

The suite should get better every time something goes wrong. Every user complaint, every bad output someone screenshots in Slack, becomes a case with the correct behaviour recorded.

Do this consistently for six months and you end up with something genuinely valuable: an executable specification of what your feature is supposed to do, written by reality rather than by a product doc.

The second-order effects

Teams that measure quality behave differently. Prompt changes stop being scary, so they happen more often. Model upgrades become a decision with evidence. You run the suite against the new model, look at the deltas, and know. Cost optimisation becomes safe, because you can drop to a smaller model on the easy path and prove that quality held.

And the conversation with stakeholders changes completely. "It feels better" becomes "accuracy went from 0.91 to 0.94 and cost per request dropped 38%." That is a sentence you can put in a board update.

Start small today

Take twenty real inputs. Write down what a good answer looks like for each. Write the simplest grader that would catch the failure you are most afraid of. Run it. Commit it.

Twenty cases in a repository beats a perfect evaluation platform you will build next quarter.