QUIZ
Applied evals
14 questions on Applied evals, from 24 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering, ML Engineering.
Question 1 of 14
An agent scores 59% when each task is run once. Why might customers see worse?
- A. The model gets tired after many tasks in a row
- B. Public benchmarks are always far too easy for agents
- C. 59% is the worst case, so customers will usually see much better results
- D. Customers need it every time; pass^k drops if it's inconsistent
View answer
✓ D. Customers need it every time; pass^k drops if it's inconsistent
Reliability across repeated runs, not one lucky pass, is what matters in production.
Question 2 of 14
Which metric tells you whether an agent used the right tool with valid arguments?
- A. The number of users who signed up that week
- B. Tokens generated per second during the task
- C. Tool-call accuracy: right tool, valid arguments
- D. The length of the agent's final answer to the user
View answer
✓ C. Tool-call accuracy: right tool, valid arguments
Measure each level: tool calls, the trajectory, the whole task, and production outcomes.
Lesson: Tool-Call Evals: Selection, Arguments, Trajectories and Effects
Question 3 of 14
Which is a known bias of LLM-as-a-judge comparisons?
- A. Position bias: favouring the first (or second) answer
- B. Agreeing perfectly with human ratings
- C. Refusing to judge pairs that look similar
- D. Always preferring the shorter of two answers, whatever they say
View answer
✓ A. Position bias: favouring the first (or second) answer
Swap the order and average, and check agreement with human ratings before trusting a judge's win rate.
Lesson: LLM Judges: Pairwise Preferences and Pass/Fail Checks
Question 4 of 14
Your summariser switches between bullets and paragraphs from run to run. What should you do first?
- A. Raise the temperature
- B. Fine-tune the model
- C. Add one line to the prompt saying which format to use
- D. Build an LLM judge for format
View answer
✓ C. Add one line to the prompt saying which format to use
It's a specification failure: the prompt never said. Fix ambiguity first and build evals only for what remains.
Question 5 of 14
Which check is best suited to run inline as a guardrail on every answer?
- A. Human review
- B. A weekly error analysis
- C. A regex for card numbers with a safe fallback
- D. An LLM judge for helpfulness
View answer
✓ C. A regex for card numbers with a safe fallback
Guardrails sit in the critical path, so they must take milliseconds and rarely misfire, which usually means code.
Lesson: Choosing the Right Eval: Ship Checks, Regression Tests, Monitors and Guardrails
Question 6 of 14
Why generate synthetic test queries through tuples of dimensions instead of asking an LLM for queries directly?
- A. LLMs can't write queries
- B. It's cheaper
- C. Direct prompts repeat the same easy request; tuples force coverage of features, user types and scenarios
- D. Tuples remove the need for real traces
View answer
✓ C. Direct prompts repeat the same easy request; tuples force coverage of features, user types and scenarios
Defining dimensions, then tuples, then queries in a separate step keeps the test set varied.
Lesson: Building Your First Eval Set: Real Traces and Synthetic Queries
Question 7 of 14
During open coding, which failure should you note for each trace?
- A. The last step's error
- B. Only hallucinations
- C. The first, most upstream failure
- D. Every problem you see
View answer
✓ C. The first, most upstream failure
Early mistakes cascade into later symptoms; noting the first keeps counts honest and points to the root cause.
Question 8 of 14
A judge has TPR 95% and TNR 60%, and reports an 85% pass rate on new traffic. What's the corrected rate?
- A. About 60%
- B. About 82%
- C. About 92%
- D. About 85%
View answer
✓ B. About 82%
(0.85 + 0.60 − 1) / (0.95 + 0.60 − 1) ≈ 0.82. A low TNR lets real failures through, so the raw rate is too rosy.
Lesson: Can You Trust Your Judge? TPR, TNR and Corrected Pass Rates
Question 9 of 14
Two annotators agree on 84% of labels, but Cohen's kappa is 0.24. Why?
- A. Percent agreement ignores ties
- B. Kappa is broken for binary labels
- C. Most answers pass, so much of the agreement is expected by chance
- D. They labelled different traces
View answer
✓ C. Most answers pass, so much of the agreement is expected by chance
Kappa subtracts chance agreement, which is high when both annotators pass almost everything.
Lesson: Labelling with a Team: Rubrics, Cohen's Kappa and Alignment
Question 10 of 14
In τ²-bench, letting the agent act alone instead of guiding a user who acts raised pass rates by about 20 points. What does that tell you?
- A. The user simulator is broken
- B. Dual control makes tasks impossible
- C. Coordinating with a user, not reasoning about the task, is where agents lose points
- D. Agents should never talk to users
View answer
✓ C. Coordinating with a user, not reasoning about the task, is where agents lose points
Running a no-user ablation separates reasoning failures from communication and coordination failures.
Question 11 of 14
A coding agent passes tasks whose tests contradict the specification. What is it most likely doing?
- A. Solving them cleverly
- B. Using a better model
- C. Gaming the tests, for example by editing them or special-casing inputs
- D. Running the tests twice
View answer
✓ C. Gaming the tests, for example by editing them or special-casing inputs
No correct solution can pass contradictory tests, so a pass means cheating. Protect tests and give the agent an abort option.
Question 12 of 14
Your MCP server's calls are 99% schema-valid, yet agents often pick another server's tool. What should your eval add?
- A. A longer tool description for every tool
- B. Tasks that don't name the tools, run with distractor servers connected
- C. A faster server
- D. More schema checks
View answer
✓ B. Tasks that don't name the tools, run with distractor servers connected
Format is largely solved; choosing the right tool among many is the hard part, so test selection with distractors.
Lesson: Evaluating an MCP Server: Tool Choice, Descriptions and Distractors
Question 13 of 14
A golden-set pass rate falls from 94% to 90% on 150 cases. What's the best next step?
- A. Double the temperature and rerun
- B. Block the release automatically
- C. Ignore it, since it's within noise
- D. Read which cases flipped; any critical case failing is a regression
View answer
✓ D. Read which cases flipped; any critical case failing is a regression
On small sets a 4-point drop may be noise on average, but specific critical cases matter regardless.
Lesson: Evals in CI: Golden Sets, Regression Gates and Model Upgrades
Question 14 of 14
An injection detector cuts attack success from 58% to 8%. Why might you still not ship it?
- A. Detectors are always slow
- B. It only works on GPT-4o
- C. Attack success should be 0%
- D. It may block so much legitimate content that normal tasks fail far more often
View answer
✓ D. It may block so much legitimate content that normal tasks fail far more often
In AgentDojo it cut normal task success from 69% to 41%. Always report utility with attack success.
Lesson: Safety and Red-Team Evals: Jailbreaks, Injection and Tool Poisoning