AIML Rounds

QUIZ

Applied evals

14 questions on Applied evals, from 24 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering, ML Engineering.

  1. Question 1 of 14

    An agent scores 59% when each task is run once. Why might customers see worse?

    1. A. The model gets tired after many tasks in a row
    2. B. Public benchmarks are always far too easy for agents
    3. C. 59% is the worst case, so customers will usually see much better results
    4. D. Customers need it every time; pass^k drops if it's inconsistent
    View answer

    ✓ D. Customers need it every time; pass^k drops if it's inconsistent

    Reliability across repeated runs, not one lucky pass, is what matters in production.

    Lesson: Agent Evals: End States, pass^k and Cost

  2. Question 2 of 14

    Which metric tells you whether an agent used the right tool with valid arguments?

    1. A. The number of users who signed up that week
    2. B. Tokens generated per second during the task
    3. C. Tool-call accuracy: right tool, valid arguments
    4. D. The length of the agent's final answer to the user
    View answer

    ✓ C. Tool-call accuracy: right tool, valid arguments

    Measure each level: tool calls, the trajectory, the whole task, and production outcomes.

    Lesson: Tool-Call Evals: Selection, Arguments, Trajectories and Effects

  3. Question 3 of 14

    Which is a known bias of LLM-as-a-judge comparisons?

    1. A. Position bias: favouring the first (or second) answer
    2. B. Agreeing perfectly with human ratings
    3. C. Refusing to judge pairs that look similar
    4. D. Always preferring the shorter of two answers, whatever they say
    View answer

    ✓ A. Position bias: favouring the first (or second) answer

    Swap the order and average, and check agreement with human ratings before trusting a judge's win rate.

    Lesson: LLM Judges: Pairwise Preferences and Pass/Fail Checks

  4. Question 4 of 14

    Your summariser switches between bullets and paragraphs from run to run. What should you do first?

    1. A. Raise the temperature
    2. B. Fine-tune the model
    3. C. Add one line to the prompt saying which format to use
    4. D. Build an LLM judge for format
    View answer

    ✓ C. Add one line to the prompt saying which format to use

    It's a specification failure: the prompt never said. Fix ambiguity first and build evals only for what remains.

    Lesson: Why LLM Apps Fail: The Three Gulfs

  5. Question 5 of 14

    Which check is best suited to run inline as a guardrail on every answer?

    1. A. Human review
    2. B. A weekly error analysis
    3. C. A regex for card numbers with a safe fallback
    4. D. An LLM judge for helpfulness
    View answer

    ✓ C. A regex for card numbers with a safe fallback

    Guardrails sit in the critical path, so they must take milliseconds and rarely misfire, which usually means code.

    Lesson: Choosing the Right Eval: Ship Checks, Regression Tests, Monitors and Guardrails

  6. Question 6 of 14

    Why generate synthetic test queries through tuples of dimensions instead of asking an LLM for queries directly?

    1. A. LLMs can't write queries
    2. B. It's cheaper
    3. C. Direct prompts repeat the same easy request; tuples force coverage of features, user types and scenarios
    4. D. Tuples remove the need for real traces
    View answer

    ✓ C. Direct prompts repeat the same easy request; tuples force coverage of features, user types and scenarios

    Defining dimensions, then tuples, then queries in a separate step keeps the test set varied.

    Lesson: Building Your First Eval Set: Real Traces and Synthetic Queries

  7. Question 7 of 14

    During open coding, which failure should you note for each trace?

    1. A. The last step's error
    2. B. Only hallucinations
    3. C. The first, most upstream failure
    4. D. Every problem you see
    View answer

    ✓ C. The first, most upstream failure

    Early mistakes cascade into later symptoms; noting the first keeps counts honest and points to the root cause.

    Lesson: Error Analysis: From Traces to Failure Rates

  8. Question 8 of 14

    A judge has TPR 95% and TNR 60%, and reports an 85% pass rate on new traffic. What's the corrected rate?

    1. A. About 60%
    2. B. About 82%
    3. C. About 92%
    4. D. About 85%
    View answer

    ✓ B. About 82%

    (0.85 + 0.60 − 1) / (0.95 + 0.60 − 1) ≈ 0.82. A low TNR lets real failures through, so the raw rate is too rosy.

    Lesson: Can You Trust Your Judge? TPR, TNR and Corrected Pass Rates

  9. Question 9 of 14

    Two annotators agree on 84% of labels, but Cohen's kappa is 0.24. Why?

    1. A. Percent agreement ignores ties
    2. B. Kappa is broken for binary labels
    3. C. Most answers pass, so much of the agreement is expected by chance
    4. D. They labelled different traces
    View answer

    ✓ C. Most answers pass, so much of the agreement is expected by chance

    Kappa subtracts chance agreement, which is high when both annotators pass almost everything.

    Lesson: Labelling with a Team: Rubrics, Cohen's Kappa and Alignment

  10. Question 10 of 14

    In τ²-bench, letting the agent act alone instead of guiding a user who acts raised pass rates by about 20 points. What does that tell you?

    1. A. The user simulator is broken
    2. B. Dual control makes tasks impossible
    3. C. Coordinating with a user, not reasoning about the task, is where agents lose points
    4. D. Agents should never talk to users
    View answer

    ✓ C. Coordinating with a user, not reasoning about the task, is where agents lose points

    Running a no-user ablation separates reasoning failures from communication and coordination failures.

    Lesson: Simulated Users and Test Environments for Agents

  11. Question 11 of 14

    A coding agent passes tasks whose tests contradict the specification. What is it most likely doing?

    1. A. Solving them cleverly
    2. B. Using a better model
    3. C. Gaming the tests, for example by editing them or special-casing inputs
    4. D. Running the tests twice
    View answer

    ✓ C. Gaming the tests, for example by editing them or special-casing inputs

    No correct solution can pass contradictory tests, so a pass means cheating. Protect tests and give the agent an abort option.

    Lesson: Coding Agent Evals: Hidden Tests and Test Tampering

  12. Question 12 of 14

    Your MCP server's calls are 99% schema-valid, yet agents often pick another server's tool. What should your eval add?

    1. A. A longer tool description for every tool
    2. B. Tasks that don't name the tools, run with distractor servers connected
    3. C. A faster server
    4. D. More schema checks
    View answer

    ✓ B. Tasks that don't name the tools, run with distractor servers connected

    Format is largely solved; choosing the right tool among many is the hard part, so test selection with distractors.

    Lesson: Evaluating an MCP Server: Tool Choice, Descriptions and Distractors

  13. Question 13 of 14

    A golden-set pass rate falls from 94% to 90% on 150 cases. What's the best next step?

    1. A. Double the temperature and rerun
    2. B. Block the release automatically
    3. C. Ignore it, since it's within noise
    4. D. Read which cases flipped; any critical case failing is a regression
    View answer

    ✓ D. Read which cases flipped; any critical case failing is a regression

    On small sets a 4-point drop may be noise on average, but specific critical cases matter regardless.

    Lesson: Evals in CI: Golden Sets, Regression Gates and Model Upgrades

  14. Question 14 of 14

    An injection detector cuts attack success from 58% to 8%. Why might you still not ship it?

    1. A. Detectors are always slow
    2. B. It only works on GPT-4o
    3. C. Attack success should be 0%
    4. D. It may block so much legitimate content that normal tasks fail far more often
    View answer

    ✓ D. It may block so much legitimate content that normal tasks fail far more often

    In AgentDojo it cut normal task success from 69% to 41%. Always report utility with attack success.

    Lesson: Safety and Red-Team Evals: Jailbreaks, Injection and Tool Poisoning