ML ENGINEERING · ROUND 3 OF 3
ML system design and MLOps
One design question, explored through follow-ups. The question: design the product recommendations on an e-commerce home page, and keep them working in production.
Question 1 of 8
Start with requirements and success metrics.
Answer out loud first, then open the model answer.
View model answer
I'd ask about users and catalogue size, traffic, latency limits for the page, and what the business wants: clicks, purchases, or long-term retention. I'd assume tens of millions of users, millions of items, and a budget of about 100 ms for recommendations.
Online, the main metric might be purchases or revenue per session, with guardrails such as diversity, latency and returns. Offline, I'd use ranking metrics such as recall@k and NDCG on held-out interactions, knowing they only approximate the online result.
What the interviewer listens for
- Clarifies scale, latency and business goal
- Online metric + guardrails
- Offline ranking metrics
- Knows offline ≠ online
Question 2 of 8
Sketch the architecture.
Answer out loud first, then open the model answer.
View model answer
Two stages. Candidate generation cheaply narrows millions of items to a few hundred, using sources such as a two-tower embedding model with approximate nearest-neighbour search, co-purchase lists and trending items. Then a ranking model, for example gradient-boosted trees or a neural ranker, scores those candidates with rich user, item and context features.
A final re-ranking step applies business rules: remove out-of-stock items, add diversity, respect promotions. Features come from a feature store shared by training and serving.
What the interviewer listens for
- Retrieval then ranking (then re-ranking)
- Several candidate sources
- Rich features in the ranker
- Business rules and diversity
Lessons: Design a recommendation system
Question 3 of 8
Where does the training data come from, and what can go wrong with it?
Answer out loud first, then open the model answer.
View model answer
From logged impressions and interactions: what we showed, and what users clicked or bought. The danger is a feedback loop: the model only learns about items it already chose to show, so popular items get more popular and new items never get a chance.
To counter it, I'd reserve a small exploration slice that shows varied items, log the probability each item was shown, and use it to correct for exposure. I'd also guard against position bias, since top slots get clicks regardless of quality.
What the interviewer listens for
- Logged impressions + outcomes
- Feedback loops and popularity bias
- Exploration and propensity logging
- Position bias
Question 4 of 8
How do you serve features and predictions within the latency budget?
Answer out loud first, then open the model answer.
View model answer
I'd precompute what can be precomputed: item embeddings, the ANN index and slow-changing user features in batch. Fresh signals, like what the user clicked a minute ago, come from a streaming pipeline into an online feature store.
At request time, the service fetches features, retrieves candidates, and ranks a few hundred items, keeping the ranking model small enough for the budget. The same feature definitions are used in training to avoid training/serving skew.
What the interviewer listens for
- Batch vs online vs streaming split
- Online feature store for fresh signals
- Ranking a few hundred, not millions
- Same feature definitions to avoid skew
Lessons: Batch vs Online vs Streaming Inference, Training/Serving Skew and Feature Stores
Question 5 of 8
A new ranking model looks better offline. How do you prove it's better for users?
Answer out loud first, then open the model answer.
View model answer
Offline gains often don't carry over, so I'd test online. For ranking, interleaving is a fast first filter: one user sees a list mixing both models' results, and clicks show which model they prefer, needing far fewer users than an A/B test.
Promising models then go to a proper A/B test on the business metric with guardrails, with a sample size planned in advance and enough run time to see past novelty effects.
What the interviewer listens for
- Offline ≠ online
- Interleaving for fast ranker comparison
- A/B test on business metric + guardrails
- Planned sample size; novelty
Lessons: Test Rankers Faster with Interleaving, Hypothesis Tests, p-values & A/B Testing
Question 6 of 8
How do you deploy the new model safely?
Answer out loud first, then open the model answer.
View model answer
First shadow mode: the new model runs on live traffic but its output isn't shown, so I can compare predictions, latency and errors. Then a canary: a small share of users gets it while I watch metrics and guardrails, then a gradual ramp-up.
Every model is versioned in a registry with its data, code and metrics, so rolling back is one switch rather than a rebuild.
What the interviewer listens for
- Shadow → canary → ramp-up
- Compare latency and errors as well as quality
- Model registry with versions
- Fast rollback
Lessons: Deployment Patterns: Shadow, Canary, A/B and Rollback, Experiment Tracking, Reproducibility and the Model Registry
Question 7 of 8
After launch, how do you know the model is degrading?
Answer out loud first, then open the model answer.
View model answer
Business metrics such as clicks and purchases come in quickly here, so I'd watch them by segment. I'd also monitor input feature drift, the distribution of scores, and data quality: missing features, schema changes, stale tables.
Where labels are slow, as in fraud, I'd watch drift and estimate performance from calibrated confidence until labels arrive. Alerts should go to an owner with a runbook, not just a dashboard.
What the interviewer listens for
- Business metrics by segment
- Feature and prediction drift; data quality
- Label-free estimation when labels are slow
- Alerts with owners and runbooks
Lessons: Drift Detection and Monitoring, Monitoring Without Labels: Estimating Performance
Question 8 of 8
How often do you retrain, and what stops a bad retrain from shipping?
Answer out loud first, then open the model answer.
View model answer
It depends on how fast things change: catalogues and tastes shift, so daily or weekly retraining is common, and embeddings may refresh more often than the ranker. I'd pick the cadence by measuring how quickly an old model's performance decays.
Every retrain passes automatic gates: data validation, offline metrics compared with the current model on the same holdout, and checks on key segments. Only then does it go through shadow or canary, with automatic rollback if online metrics drop.
What the interviewer listens for
- Cadence set by measured decay
- Data validation before training
- Compare with the current model; segment checks
- Staged rollout and rollback after retrain
Lessons: Retraining Pipelines and Continuous Training, MLOps Practice