AIML Rounds

QUIZ

MLOps

8 questions on MLOps, from 9 lessons. Pick an answer in your head, then open it.

On the roadmap for: ML Engineering.

  1. Question 1 of 8

    A model looks great offline but fails online. What is training/serving skew?

    1. A. The model is too big for the serving hardware
    2. B. Users changed their behaviour overnight
    3. C. Features computed differently in training and production
    4. D. The serving GPUs are slower than the ones used for training
    View answer

    ✓ C. Features computed differently in training and production

    Computing features once, from shared definitions such as a feature store, keeps them consistent.

    Lesson: Training/Serving Skew and Feature Stores

  2. Question 2 of 8

    When are batch (precomputed) predictions a good fit?

    1. A. When each request needs fresh, live context
    2. B. Never: batch predictions are always stale
    3. C. Only for image models
    4. D. When predictions can be computed ahead of time
    View answer

    ✓ D. When predictions can be computed ahead of time

    Batch is cheap and simple; online inference is for decisions that depend on live data.

    Lesson: Batch vs Online vs Streaming Inference

  3. Question 3 of 8

    What is a canary deployment?

    1. A. Testing the new model offline only, on a held-out set from last month
    2. B. Running the new model on no live traffic at all
    3. C. Sending a small share of traffic to the new model first
    4. D. Deleting the old model once the new one is trained
    View answer

    ✓ C. Sending a small share of traffic to the new model first

    If metrics drop, you roll back with little impact. Shadow mode runs the model without serving its outputs.

    Lesson: Deployment Patterns: Shadow, Canary, A/B and Rollback

  4. Question 4 of 8

    What does a model registry help you do?

    1. A. Training models faster by using more GPUs
    2. B. Labelling new training data automatically
    3. C. Tracking model versions with data and code
    4. D. Picking the best hyperparameters for you
    View answer

    ✓ C. Tracking model versions with data and code

    Without it, rebuilding last month's model or rolling back quickly is guesswork.

    Lesson: Experiment Tracking, Reproducibility and the Model Registry

  5. Question 5 of 8

    Labels arrive weeks late. What can you monitor today to catch problems?

    1. A. Nothing useful until the labels arrive
    2. B. Only the loss on the original training set
    3. C. The size of the model file on disk
    4. D. Drift in input features and predictions
    View answer

    ✓ D. Drift in input features and predictions

    Drift in inputs or outputs is an early warning, although not every drift hurts accuracy.

    Lesson: Drift Detection and Monitoring

  6. Question 6 of 8

    How can you estimate a model's accuracy before any labels arrive?

    1. A. Use its calibrated confidence scores to estimate it
    2. B. Assume accuracy is unchanged since the day of training
    3. C. Ask a sample of users whether it's right
    4. D. You can't, until the labels arrive
    View answer

    ✓ A. Use its calibrated confidence scores to estimate it

    If probabilities are well calibrated, their average tells you the expected accuracy, a useful early check.

    Lesson: Monitoring Without Labels: Estimating Performance

  7. Question 7 of 8

    What should stop a bad automatic retrain from reaching production?

    1. A. Retraining more often so bad models don't last long
    2. B. A gate comparing it with the current model first
    3. C. Nothing; automatic retrains are always improvements
    4. D. Using a bigger model architecture for every retrain
    View answer

    ✓ B. A gate comparing it with the current model first

    Automated retraining needs automated checks, or a data problem ships straight to users.

    Lesson: Retraining Pipelines and Continuous Training

  8. Question 8 of 8

    A recommender retrained weekly on its own clicks keeps showing the same few items. Why?

    1. A. The model is too small to learn more items
    2. B. Users dislike variety, so they click less on new items
    3. C. It only learns from what it showed: a feedback loop
    4. D. Clicks are random, so it can't learn
    View answer

    ✓ C. It only learns from what it showed: a feedback loop

    Add some exploration or a randomised slice, and correct for exposure, to get unbiased training data.

    Lesson: Feedback Loops and Unbiased Training Data