QUIZ
Evaluation metrics
10 questions on Evaluation metrics, from 7 lessons. Pick an answer in your head, then open it.
On the roadmap for: ML Engineering, Data Science.
Question 1 of 10
Fraud is 0.2% of transactions, and your model is 99.8% accurate. What's the likely problem?
- A. It may simply predict 'not fraud' for every transaction
- B. None; 99.8% accuracy means the model is excellent
- C. Accuracy is too low for a fraud model to be useful in production
- D. The data is balanced, so accuracy is misleading
View answer
✓ A. It may simply predict 'not fraud' for every transaction
Always predicting the majority class already scores 99.8%. Look at precision and recall on the fraud class.
Lesson: Classification Metrics
Question 2 of 10
What is precision?
- A. (TP + TN) / total: how many predictions were right
- B. FP / (FP + TN): how many negatives were wrongly flagged
- C. TP / (TP + FP): of everything flagged, how much was right
- D. TP / (TP + FN): of all real positives, how many were caught
View answer
✓ C. TP / (TP + FP): of everything flagged, how much was right
Precision asks how trustworthy a flag is. Recall, TP / (TP + FN), asks how many real cases you caught.
Lesson: Classification Metrics
Question 3 of 10
Why use the F1 score rather than the average of precision and recall?
- A. It weights recall twice as much as precision
- B. As a harmonic mean, it stays low if either one is low
- C. It also counts the true negatives, which a simple average leaves out
- D. It's easier to compute than a simple average
View answer
✓ B. As a harmonic mean, it stays low if either one is low
Precision 1.0 with recall 0.01 averages to about 0.5, but F1 ≈ 0.02, which reflects that the model is useless.
Lesson: Classification Metrics
Question 4 of 10
Positives are very rare. Why is PR-AUC often more informative than ROC-AUC?
- A. ROC-AUC ignores the positive class completely
- B. ROC-AUC needs calibrated probabilities to be meaningful
- C. PR-AUC is always the higher number, so differences are easier to see on it
- D. With many negatives, FPR stays tiny while false alarms swamp the hits
View answer
✓ D. With many negatives, FPR stays tiny while false alarms swamp the hits
1,000 false alarms out of a million negatives is an FPR of 0.1%, yet precision may be terrible. PR-AUC exposes it.
Lesson: ROC-AUC vs PR-AUC
Question 5 of 10
A missed fraud costs 100 times as much as reviewing a false alarm. What threshold on the predicted probability minimises expected cost?
- A. 0.5, the usual default
- B. About 0.01
- C. About 0.1
- D. 0.99, to be sure before flagging
View answer
✓ B. About 0.01
Flag when p · 100 > (1 − p) · 1, which gives p > 1/101 ≈ 0.01. Costs, not 0.5, set the threshold.
Question 6 of 10
After a model change, RMSE went up sharply but MAE barely moved. What most likely happened?
- A. The model got better on most cases, so MAE held
- B. The two metrics are always independent of each other
- C. A few predictions got much worse (large errors)
- D. Every prediction got slightly worse by the same amount
View answer
✓ C. A few predictions got much worse (large errors)
RMSE squares errors, so a handful of big misses dominate it, while MAE weighs every error equally.
Lesson: Regression Metrics
Question 7 of 10
What does NDCG reward in a ranked list?
- A. How many items the list contains in total
- B. Only whether any relevant item appears anywhere in the list at all
- C. How diverse the items in the list are
- D. Relevant items near the top, with less credit lower down
View answer
✓ D. Relevant items near the top, with less credit lower down
Gains are discounted by log position and normalised by the ideal order, so 1.0 means the best possible ranking.
Lesson: Ranking Metrics
Question 8 of 10
You're evaluating the retriever in a RAG system that passes 5 chunks to the LLM. Which metric matters most?
- A. RMSE of the similarity scores
- B. Accuracy of the answers against reference answers
- C. Average length of the retrieved chunks
- D. Recall@5: did the needed chunk reach the top 5?
View answer
✓ D. Recall@5: did the needed chunk reach the top 5?
If the right chunk isn't retrieved, the generator can't use it. Order within the 5 matters less.
Lesson: Ranking Metrics
Question 9 of 10
A model ranks customers perfectly, but its probabilities are off: cases scored 0.8 happen only 40% of the time. What is wrong?
- A. Calibration
- B. Recall
- C. Ranking
- D. Precision
View answer
✓ A. Calibration
Good ranking (AUC) doesn't mean trustworthy probabilities. Check reliability plots or Brier score, and recalibrate if needed.
Lesson: Probability Metrics
Question 10 of 10
For a cancer screening test, missing a real case is far worse than a false alarm. Which metric should you prioritise?
- A. Recall (sensitivity)
- B. Precision (positive predictive value)
- C. Specificity (true negative rate)
- D. Accuracy (share of all predictions right)
View answer
✓ A. Recall (sensitivity)
Recall measures how many real cases you catch. False alarms get filtered by follow-up tests.
Lesson: Choosing the Right Metric