AIML Rounds RoadmapMocksJobs
Concept15 minBefore this:Why LLM Apps Fail: The Three Gulfs

Choosing the Right Eval: Ship Checks, Regression Tests, Monitors and Guardrails

Should this check run in CI, on live traffic, or before every answer reaches the user?

The situation

Your team built an LLM judge that checks answers for invented policy details. It agrees with experts 93% of the time. The product manager wants it on every answer before users see it; engineering wants it in CI only. The interviewer asks:

Should this check run in CI, on live traffic, or before every answer reaches the user?

Why it matters

Teams often build one big "eval" and then don't know what to do with its number. Is 82% good? Should it block a release? Can it run on every request? Interviewers probe exactly this: where each check lives, and why. The answer depends on the check's job and on how fast and cheap it is.

The idea

Think of quality checks in a bakery. A metal detector checks every loaf on the line in a split second. A taster samples a few loaves each hour. A new recipe gets a full panel before launch. Each check does a different job, and none could do the others'.

GuardrailEvery answer,millisecondsMostly codeCIFixed cases,every changeNear 100%Monitor1–5% sample,asyncJudges fit
Latency and cost decide where a check lives.

Play

Try to break it. Put the LLM judge inline with a 200 ms budget. Then raise traffic.

Where can this check run?

Latency and cost decide placement

Kind of check

as a guardrail
–
cost if run on every answer
–
cost on a 2% sample
–

A regex fits any latency budget, so it can block a leaked card number before the user sees it. An LLM judge at about 2.5 seconds doesn't fit a 200 ms budget, and on all 10,000 daily answers it would cost about $200 a day. On a 2% sample it costs $4 and still tells you the failure rate. That's why judges usually run after the fact, and guardrails are mostly cheap code.

Now say why: why does sampling make the judge affordable without losing the failure rate?

How it works

In one breath

An eval is a check with a job. A capability eval starts with a low pass rate and shows progress. A regression eval should pass nearly 100% and guards what already works. A monitor runs on a sample of live traffic after the fact. A guardrail runs on every answer before the user sees it, so it must take milliseconds and rarely fire by mistake. Code checks fit everywhere; LLM judges take seconds and cost money, so they belong in CI and sampled monitoring; humans go where judgement and new failures need finding.

Jobs and kinds

The jobs

JobQuestionHealthy pass rateRuns
CapabilityCan we do this at all yet?low at first, a hill to climbduring development
RegressionDid this change break something that worked?close to 100%CI, on every prompt, model or tool change
MonitorHow often does this failure happen in production?tracked over time, with error barsasync, on a 1–5% sample of traffic
GuardrailShould this answer reach the user?fires rarely, almost never by mistakeinline, on every answer
MigrationIs the new model at least as good on our cases?compared side by sidebefore switching models

Anthropic's agent-evals guide describes capability evals "graduating" into the regression suite once they pass reliably. An eval stuck at 100% still catches regressions but no longer tells you how to improve.

The kinds of check

  • Code: schema validation, regex, running generated SQL, comparing to a reference. Fast, cheap, reproducible, but brittle on open-ended text.
  • LLM judge: a model grades one failure mode with a pass/fail answer. Handles nuance, but takes seconds, costs money and needs checking against human labels.
  • Human: the gold standard, and slow. Best for finding new failure types and labelling the data judges are checked against.

Two more distinctions

Reference-based checks compare with a known right answer, which suits CI's fixed test cases. Reference-free checks test a property ("cites a source", "no personal data"), which suits live traffic where no right answer exists. And absolute evals ask "good enough to ship?", while comparative ones ask "is B better than A?" with pairwise judging or an A/B test.

In practice

In practice

  • Give every eval a job: capability, regression, monitor, guardrail or migration, and decide what its number will trigger.
  • Guardrails are mostly code: they must be fast and rarely wrong, since a false alarm blocks a good answer.
  • Judges run async: on CI's fixed set and on a small sample of live traffic.
  • Use humans to discover: new failure types, and the labels that keep judges honest.
  • Graduate evals: once a capability eval passes reliably, move it into the regression suite.

The answer that sounds right

"Run the judge on every answer to be safe"

It sounds safer, but it adds seconds to every reply and costs far more, and a 2% sample gives the same failure rate. Keep inline checks fast and cheap, and run judges after the fact.

Say it

Should this check run in CI, on live traffic, or before every answer reaches the user?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

The interviewer pushes further

Follow-up 1 of 3

Can an LLM judge be a guardrail?

Answer it out loud, then check.

Sometimes, for slow, high-stakes flows where a few seconds are acceptable, or with a small fast classifier. For chat, a 2–3 second judge on every answer is usually too slow, so guardrails are mostly code.

Follow-up 2 of 3

Your eval passes 100%. Is that good?

Answer it out loud, then check.

For a regression suite, yes. For a capability eval, it means the eval no longer tells you how to improve, so add harder cases.

Follow-up 3 of 3

When do you need a comparative eval rather than an absolute one?

Answer it out loud, then check.

When choosing between two versions, prompts or models. Pairwise judging or an A/B test answers "which is better" more reliably than two separate scores.

I'd give each eval a job. Capability evals start low and show progress. Regression evals should pass almost 100% and run in CI on every prompt, model or tool change. Monitors run judges on a small sample of live traffic, asynchronously. Guardrails run inline on every answer, so they must take milliseconds and rarely fire wrongly, which usually means code like regex and schema checks. LLM judges take seconds and cost money, so they belong in CI and sampled monitoring, and humans find new failures and label data for the judges.

Key terms and sources

The different jobs an eval can do (capability, regression, monitoring, guardrail, model migration), the kinds of check (code, LLM judge, human), and how latency and cost decide where each one can run.

Sources

Explanations and examples are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Many recent papers are preprints, and benchmark scores change quickly.