Evaluating LLM Apps
Why can a correct answer still fail your evaluation?
The situation
A customer asks how much refund they get for a late order. The right answer needs two facts: 10%, capped at ₹150. The assistant says exactly that, so a colleague says ship it. But the chunk with the cap was never retrieved. The interviewer asks:
Why can a correct answer still fail your evaluation?
Running example: a customer asks "How much refund do I get for a late order?" The reference answer needs two facts from the policy: the refund is 10%, capped at ₹150.
Open-ended outputs, many moving parts
An LLM app's answer depends on the prompt, the model version, the retriever, the index, the chunking and the documents. A change to any of them can fix one case and silently break ten others. Without a repeatable evaluation, every change is a guess.
The idea
Think of a student who answers a comprehension question correctly without reading the passage. They get the mark today. But when the passage changes, they'll confidently give the old answer. So you check that the answer came from the passage, as well as that it's right.
Play
Try to break it. Start with good retrieval and the right answer. Then add the invented voucher. Then switch to noisy retrieval, and go back to the right answer.
"The answer was right, ship it"
Good retrieval: chunks "10% refund for orders over 30 min late", "late refunds capped at ₹150", "wallet refunds are instant". Noisy: "Gold costs ₹149/month", the 10% chunk, "tips go to the partner". Simplified metrics: precision = relevant retrieved chunks ÷ retrieved; recall = needed facts retrieved ÷ needed; faithfulness = answer claims found in the context ÷ answer claims; correctness = needed facts in the answer ÷ needed.
With good retrieval and the right answer, everything is 1.00 except precision (0.67: one retrieved chunk was irrelevant, harmless). If the model adds "and a free dessert voucher", correctness stays 1.00 but faithfulness drops to 0.67: an invented promise that a correctness-only check misses. Now the trap: with noisy retrieval (the cap chunk missing), the model still answers "10%, capped at ₹150", so correctness is 1.00. But context recall is 0.50 and faithfulness 0.50: half the answer came from the model's memory, not the policy. It's right today; when the cap changes to ₹100, it will confidently say ₹150. If it answers only "10%", faithfulness is 1.00 but correctness falls to 0.50: the metrics together tell you where to fix.
Now say why: with noisy retrieval the answer is still correct, so what's actually wrong?
How it works
In one breath
"it looked good on five examples" is how LLM apps fail in production. Start a golden set early: a few hundred real inputs with the expected answer or behaviour, including hard and adversarial cases. For RAG, score retrieval and generation separately: context recall (did we retrieve everything the answer needs?), context precision (how much of what we retrieved is relevant?), faithfulness (is every claim in the answer supported by the retrieved context?) and answer correctness (does it match the reference?). A correct answer can still be unfaithful, a sign the model is answering from memory, which will fail when memory is wrong. Use code checks wherever possible (JSON valid, refund within policy), LLM judges validated against humans for the rest, run the suite on every prompt, model or index change, and watch online signals (escalations, thumbs-down, resolution rate).What you'll be able to do after this: design an eval suite for a RAG app, interpret its metrics, and set up regression testing.Component metrics for RAG
| Metric | Question it answers | Points to |
|---|---|---|
| Context recall | Were all facts the answer needs retrieved? | retriever, chunking, query |
| Context precision | How much of the retrieved context is relevant (and ranked first)? | ranking, k, re-ranker |
| Faithfulness | Is every claim in the answer supported by the context? | generator, prompt (hallucination) |
| Answer relevance / correctness | Does it answer the question, and match the reference? | whole pipeline |
RAGAS (Es et al. 2024) computes these with an LLM: it splits answers into claims and checks each against the context. Retrieval metrics can also be computed exactly from labelled relevant chunks (recall@k, MRR; retrieval lesson).
In practice
An evaluation process that works
- Build the golden set from real traffic (and add every production failure to it), with expected answers or rubrics; include "should refuse" and "should escalate" cases.
- Look at the data: read failures by hand and group them (retrieval miss, wrong policy version, format error, tone) before writing metrics; Shankar et al. (2024) found evaluation criteria themselves change once people see real outputs.
- Prefer deterministic checks (schema valid, refund ≤ cap, cites a source, no PII); use LLM judges for faithfulness and helpfulness, validated against a few hundred human labels.
- Regression-test in CI: run the suite on every prompt, model, retriever or index change; block releases that regress, and report confidence intervals (error bars).
- Online: track escalation rate, thumbs-down, re-asks, resolution, cost and latency; A/B test big changes; sample conversations for review.
Evaluate each component, on real cases, on every change
Separate retrieval metrics from generation metrics so failures point to their cause, check faithfulness as well as correctness, and make the golden set a regression test that grows with every bug.
The answer that sounds right
"We tested it on a few prompts"
It sounds right, since the answers looked fine. But a change to any part of the pipeline can fix one case and silently break ten others. You need a versioned golden set, run on every prompt, model or index change. And faithful answers can still be unhelpful or incomplete, so check relevance and read the complaints too.
Say it
Why can a correct answer still fail your evaluation?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
Faithfulness is high but users complain. What's missing?
Answer it out loud, then check.
Faithful answers can still be unhelpful or incomplete: check answer relevance, context recall (maybe the needed facts were never retrieved), tone and format, and read the complaints.
Follow-up 2 of 3
How many examples does the golden set need?
Answer it out loud, then check.
Start with 50–100 to catch obvious regressions, grow to several hundred for decisions; for ±3-point precision on a rate you need around 1,000.
Follow-up 3 of 3
Can you trust an LLM judge for faithfulness?
Answer it out loud, then check.
After validating it against human labels on your data, with a different model family from the generator, and with claim-level checks rather than one overall score.
The 30-second answer
I'd build a golden set of a few hundred real questions with reference answers and expected behaviours, including refusals and escalations, and grow it with every production failure. For RAG I evaluate retrieval and generation separately: context recall and precision, faithfulness of each claim to the retrieved context, and answer correctness, because a correct but unfaithful answer means the model is relying on memory. I use code checks wherever I can, LLM judges validated against human labels for the rest, run the suite in CI on every prompt, model or index change, and monitor online signals like escalations, thumbs-down and resolution rate.
Go further
What not to say
- "We tested it on a few prompts." Use a versioned golden set.
- "The answer was correct, so retrieval is fine." Check faithfulness and context recall.
Key terms and sources
How to measure whether an LLM application works: building a golden set, evaluating retrieval and generation separately (context precision and recall, faithfulness, answer correctness), code checks vs LLM judges, regression testing every change, and online signals in production · built from Es et al. (RAGAS, EACL 2024), Saad-Falcon et al. (ARES, NAACL 2024), Zheng et al. (LLM-as-a-Judge, NeurIPS 2023), Liang et al. (HELM, 2022), Shankar et al. (Who Validates the Validators?, UIST 2024) and the OpenAI and Anthropic evaluation guides · checked 29 Oct 2026
Sources
- S. Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation", EACL 2024 (demo), arXiv:2309.15217.
- J. Saad-Falcon et al., "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems", NAACL 2024, arXiv:2311.09476.
- L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023, arXiv:2306.05685.
- S. Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences", UIST 2024, arXiv:2404.12272.
- P. Liang et al., "Holistic Evaluation of Language Models" (HELM), 2022, arXiv:2211.09110; OpenAI and Anthropic evaluation guides.