AIML Rounds

QUIZ

Evaluation metrics

10 questions on Evaluation metrics, from 7 lessons. Pick an answer in your head, then open it.

On the roadmap for: ML Engineering, Data Science.

  1. Question 1 of 10

    Fraud is 0.2% of transactions, and your model is 99.8% accurate. What's the likely problem?

    1. A. It may simply predict 'not fraud' for every transaction
    2. B. None; 99.8% accuracy means the model is excellent
    3. C. Accuracy is too low for a fraud model to be useful in production
    4. D. The data is balanced, so accuracy is misleading
    View answer

    ✓ A. It may simply predict 'not fraud' for every transaction

    Always predicting the majority class already scores 99.8%. Look at precision and recall on the fraud class.

    Lesson: Classification Metrics

  2. Question 2 of 10

    What is precision?

    1. A. (TP + TN) / total: how many predictions were right
    2. B. FP / (FP + TN): how many negatives were wrongly flagged
    3. C. TP / (TP + FP): of everything flagged, how much was right
    4. D. TP / (TP + FN): of all real positives, how many were caught
    View answer

    ✓ C. TP / (TP + FP): of everything flagged, how much was right

    Precision asks how trustworthy a flag is. Recall, TP / (TP + FN), asks how many real cases you caught.

    Lesson: Classification Metrics

  3. Question 3 of 10

    Why use the F1 score rather than the average of precision and recall?

    1. A. It weights recall twice as much as precision
    2. B. As a harmonic mean, it stays low if either one is low
    3. C. It also counts the true negatives, which a simple average leaves out
    4. D. It's easier to compute than a simple average
    View answer

    ✓ B. As a harmonic mean, it stays low if either one is low

    Precision 1.0 with recall 0.01 averages to about 0.5, but F1 ≈ 0.02, which reflects that the model is useless.

    Lesson: Classification Metrics

  4. Question 4 of 10

    Positives are very rare. Why is PR-AUC often more informative than ROC-AUC?

    1. A. ROC-AUC ignores the positive class completely
    2. B. ROC-AUC needs calibrated probabilities to be meaningful
    3. C. PR-AUC is always the higher number, so differences are easier to see on it
    4. D. With many negatives, FPR stays tiny while false alarms swamp the hits
    View answer

    ✓ D. With many negatives, FPR stays tiny while false alarms swamp the hits

    1,000 false alarms out of a million negatives is an FPR of 0.1%, yet precision may be terrible. PR-AUC exposes it.

    Lesson: ROC-AUC vs PR-AUC

  5. Question 5 of 10

    A missed fraud costs 100 times as much as reviewing a false alarm. What threshold on the predicted probability minimises expected cost?

    1. A. 0.5, the usual default
    2. B. About 0.01
    3. C. About 0.1
    4. D. 0.99, to be sure before flagging
    View answer

    ✓ B. About 0.01

    Flag when p · 100 > (1 − p) · 1, which gives p > 1/101 ≈ 0.01. Costs, not 0.5, set the threshold.

    Lesson: Thresholds & Misclassification Costs

  6. Question 6 of 10

    After a model change, RMSE went up sharply but MAE barely moved. What most likely happened?

    1. A. The model got better on most cases, so MAE held
    2. B. The two metrics are always independent of each other
    3. C. A few predictions got much worse (large errors)
    4. D. Every prediction got slightly worse by the same amount
    View answer

    ✓ C. A few predictions got much worse (large errors)

    RMSE squares errors, so a handful of big misses dominate it, while MAE weighs every error equally.

    Lesson: Regression Metrics

  7. Question 7 of 10

    What does NDCG reward in a ranked list?

    1. A. How many items the list contains in total
    2. B. Only whether any relevant item appears anywhere in the list at all
    3. C. How diverse the items in the list are
    4. D. Relevant items near the top, with less credit lower down
    View answer

    ✓ D. Relevant items near the top, with less credit lower down

    Gains are discounted by log position and normalised by the ideal order, so 1.0 means the best possible ranking.

    Lesson: Ranking Metrics

  8. Question 8 of 10

    You're evaluating the retriever in a RAG system that passes 5 chunks to the LLM. Which metric matters most?

    1. A. RMSE of the similarity scores
    2. B. Accuracy of the answers against reference answers
    3. C. Average length of the retrieved chunks
    4. D. Recall@5: did the needed chunk reach the top 5?
    View answer

    ✓ D. Recall@5: did the needed chunk reach the top 5?

    If the right chunk isn't retrieved, the generator can't use it. Order within the 5 matters less.

    Lesson: Ranking Metrics

  9. Question 9 of 10

    A model ranks customers perfectly, but its probabilities are off: cases scored 0.8 happen only 40% of the time. What is wrong?

    1. A. Calibration
    2. B. Recall
    3. C. Ranking
    4. D. Precision
    View answer

    ✓ A. Calibration

    Good ranking (AUC) doesn't mean trustworthy probabilities. Check reliability plots or Brier score, and recalibrate if needed.

    Lesson: Probability Metrics

  10. Question 10 of 10

    For a cancer screening test, missing a real case is far worse than a false alarm. Which metric should you prioritise?

    1. A. Recall (sensitivity)
    2. B. Precision (positive predictive value)
    3. C. Specificity (true negative rate)
    4. D. Accuracy (share of all predictions right)
    View answer

    ✓ A. Recall (sensitivity)

    Recall measures how many real cases you catch. False alarms get filtered by follow-up tests.

    Lesson: Choosing the Right Metric