QUIZ
MLOps
8 questions on MLOps, from 9 lessons. Pick an answer in your head, then open it.
On the roadmap for: ML Engineering.
Question 1 of 8
A model looks great offline but fails online. What is training/serving skew?
- A. The model is too big for the serving hardware
- B. Users changed their behaviour overnight
- C. Features computed differently in training and production
- D. The serving GPUs are slower than the ones used for training
View answer
✓ C. Features computed differently in training and production
Computing features once, from shared definitions such as a feature store, keeps them consistent.
Question 2 of 8
When are batch (precomputed) predictions a good fit?
- A. When each request needs fresh, live context
- B. Never: batch predictions are always stale
- C. Only for image models
- D. When predictions can be computed ahead of time
View answer
✓ D. When predictions can be computed ahead of time
Batch is cheap and simple; online inference is for decisions that depend on live data.
Question 3 of 8
What is a canary deployment?
- A. Testing the new model offline only, on a held-out set from last month
- B. Running the new model on no live traffic at all
- C. Sending a small share of traffic to the new model first
- D. Deleting the old model once the new one is trained
View answer
✓ C. Sending a small share of traffic to the new model first
If metrics drop, you roll back with little impact. Shadow mode runs the model without serving its outputs.
Lesson: Deployment Patterns: Shadow, Canary, A/B and Rollback
Question 4 of 8
What does a model registry help you do?
- A. Training models faster by using more GPUs
- B. Labelling new training data automatically
- C. Tracking model versions with data and code
- D. Picking the best hyperparameters for you
View answer
✓ C. Tracking model versions with data and code
Without it, rebuilding last month's model or rolling back quickly is guesswork.
Lesson: Experiment Tracking, Reproducibility and the Model Registry
Question 5 of 8
Labels arrive weeks late. What can you monitor today to catch problems?
- A. Nothing useful until the labels arrive
- B. Only the loss on the original training set
- C. The size of the model file on disk
- D. Drift in input features and predictions
View answer
✓ D. Drift in input features and predictions
Drift in inputs or outputs is an early warning, although not every drift hurts accuracy.
Lesson: Drift Detection and Monitoring
Question 6 of 8
How can you estimate a model's accuracy before any labels arrive?
- A. Use its calibrated confidence scores to estimate it
- B. Assume accuracy is unchanged since the day of training
- C. Ask a sample of users whether it's right
- D. You can't, until the labels arrive
View answer
✓ A. Use its calibrated confidence scores to estimate it
If probabilities are well calibrated, their average tells you the expected accuracy, a useful early check.
Question 7 of 8
What should stop a bad automatic retrain from reaching production?
- A. Retraining more often so bad models don't last long
- B. A gate comparing it with the current model first
- C. Nothing; automatic retrains are always improvements
- D. Using a bigger model architecture for every retrain
View answer
✓ B. A gate comparing it with the current model first
Automated retraining needs automated checks, or a data problem ships straight to users.
Question 8 of 8
A recommender retrained weekly on its own clicks keeps showing the same few items. Why?
- A. The model is too small to learn more items
- B. Users dislike variety, so they click less on new items
- C. It only learns from what it showed: a feedback loop
- D. Clicks are random, so it can't learn
View answer
✓ C. It only learns from what it showed: a feedback loop
Add some exploration or a randomised slice, and correct for exposure, to get unbiased training data.