QUIZ
Recommenders and ranking
16 questions on Recommenders and ranking, from 12 lessons. Pick an answer in your head, then open it.
On the roadmap for: ML Engineering, Data Science.
Question 1 of 16
How does a recommender pick a few items for a user from millions?
- A. Show the most popular items to everyone, refreshed once every hour
- B. Score every item in the catalogue with the full model
- C. In stages: fast retrieval to hundreds, then a heavier ranker
- D. Pick items at random, then learn from clicks
View answer
✓ C. In stages: fast retrieval to hundreds, then a heavier ranker
Scoring millions of items with a big model per request is too slow; retrieval then ranking keeps it fast.
Lesson: Design a recommendation system
Question 2 of 16
Why can interleaving compare two rankers with far fewer users than an A/B test?
- A. It uses only offline data, so no live users are needed
- B. It skips significance testing to save time
- C. It tests one ranker at a time on all of the users
- D. Each user sees both rankers mixed; clicks pick a winner
View answer
✓ D. Each user sees both rankers mixed; clicks pick a winner
Comparing within the same user removes between-user noise, so a preference shows up much sooner.
Question 3 of 16
You want to learn which image to show each user for each title. Why use a bandit rather than a fixed A/B test?
- A. Bandits need no data to start choosing well
- B. It shifts traffic to better images while learning
- C. Bandits are always unbiased in the end, unlike A/B tests
- D. A/B tests can't compare more than two images
View answer
✓ B. It shifts traffic to better images while learning
Keep a holdout or a proper test alongside it to prove the personalisation actually helps.
Lesson: Personalise Images with Bandits
Question 4 of 16
Why keep the keyword index when adding embedding-based product search?
- A. Embeddings blur exact tokens like model numbers; keywords keep filters
- B. Keeping both saves storage compared with embeddings alone
- C. Keyword search is always more relevant than embeddings
- D. Embeddings are too slow to search at this scale
View answer
✓ A. Embeddings blur exact tokens like model numbers; keywords keep filters
Hybrid retrieval runs both and lets the re-ranker combine them; embeddings help most on long-tail queries.
Lesson: Design Semantic Product Search
Question 5 of 16
In a feed ranker, why combine several predicted actions in a value model instead of ranking by click probability?
- A. Clicks reward regretted clickbait; weighting actions fixes it
- B. It's faster to compute than a single click prediction
- C. Click probability can't be predicted at all for brand-new posts with no history
- D. Value models don't need any training data to work
View answer
✓ A. Clicks reward regretted clickbait; weighting actions fixes it
Multi-task predictions plus tunable weights let the product balance engagement types without retraining.
Lesson: Design Feed Ranking
Question 6 of 16
You train a click model keeping only 2.5% of non-click examples. What must you do before using its predictions in the auction?
- A. Multiply every prediction by 40
- B. Recalibrate with q = p / (p + (1 − p) / w)
- C. Nothing; the probabilities are already correct
- D. Retrain the model on all the data
View answer
✓ B. Recalibrate with q = p / (p + (1 − p) / w)
Downsampling inflates predicted probabilities; the auction needs real click rates, so convert them back.
Lesson: Design Ad Click Prediction
Question 7 of 16
A marketplace adds an embedding per listing to its ranking network. Training NDCG jumps, test NDCG doesn't. Why?
- A. The embedding is too small to hold each listing's detail
- B. Few bookings per listing, so its embedding memorises noise
- C. Embeddings never work for ranking problems
- D. The test set must contain different listings
View answer
✓ B. Few bookings per listing, so its embedding memorises noise
Item embeddings need many interactions per item, as with videos or words; scarce bookings make them overfit.
Question 8 of 16
Why does a notification bandit penalise templates a user saw recently?
- A. To keep the decision logs small and cheap
- B. To save money on sending each notification
- C. Templates expire after a few days and must be rewritten
- D. Fresh messages work better; repeats get tuned out
View answer
✓ D. Fresh messages work better; repeats get tuned out
A recency penalty with a half-life, plus softmax exploration, keeps messages varied while still favouring strong templates.
Lesson: Design Notification Optimisation
Question 9 of 16
In a short-video feed, why train the ranking model online on streaming actions instead of in a daily batch?
- A. It removes the need for retrieval
- B. Batch training is impossible at scale
- C. Interests and new videos change within minutes; online training picks them up, and beat batch by 14–18% AUC in the paper's live test
- D. Online training needs no labels
View answer
✓ C. Interests and new videos change within minutes; online training picks them up, and beat batch by 14–18% AUC in the paper's live test
Fast-moving feeds reward freshness; the reported live test showed a large AUC gain from training in real time.
Question 10 of 16
Why did the short-video system use a collisionless embedding table instead of hashing IDs into a fixed table?
- A. Collisionless tables use less memory always
- B. Hashing makes different IDs share a row (7.73% of user IDs collided on a public dataset), which blurred them; collisionless tables did better
- C. Hashing is slower
- D. Hashing can't handle new IDs
View answer
✓ B. Hashing makes different IDs share a row (7.73% of user IDs collided on a public dataset), which blurred them; collisionless tables did better
Shared rows mix up the signals of different users or videos; the paper kept memory in check by admitting and expiring IDs instead.
Question 11 of 16
A job recommender upgrades its embedding model. Downstream models were trained on the old vectors. What keeps them working during the switch?
- A. Learn a linear map from new vectors to the old space
- B. Average old and new vectors
- C. Retrain everything the same day
- D. Keep both models forever
View answer
✓ A. Learn a linear map from new vectors to the old space
A new model produces a different vector space; a learned map lets old consumers read new embeddings until they're retrained.
Lesson: Design Job Recommendations
Question 12 of 16
Job matching is two-sided. Which measure reflects that?
- A. Number of jobs posted
- B. Clicks on job posts only
- C. Time on page
- D. Rewarding matches where recruiters respond (e.g. accepted messages) and capping how often one job is shown
View answer
✓ D. Rewarding matches where recruiters respond (e.g. accepted messages) and capping how often one job is shown
A match only works if both sides engage, and flooding a few postings hurts recruiters and job seekers alike.
Lesson: Design Job Recommendations
Question 13 of 16
A food-delivery app launches groceries. Most new grocery users have long restaurant histories. What does the cold-start design do?
- A. Replace the ranker with an LLM
- B. Wait until each user has 20 grocery orders
- C. Show random groceries
- D. Use an LLM to translate restaurant orders and searches into affinities over the grocery taxonomy, fed to the existing ranker as features
View answer
✓ D. Use an LLM to translate restaurant orders and searches into affinities over the grocery taxonomy, fed to the existing ranker as features
The LLM adds a signal the ranker was missing while the ranker stays fast and well understood.
Question 14 of 16
The LLM sometimes outputs grocery categories that don't exist. What's the fix?
- A. Raise the temperature
- B. Let the ranker learn to ignore them
- C. Require exact taxonomy names, validate outputs and drop anything not in the taxonomy
- D. Use a bigger model
View answer
✓ C. Require exact taxonomy names, validate outputs and drop anything not in the taxonomy
Validating against a closed list turns free text into reliable features.
Question 15 of 16
Why train one recommendation foundation model across many surfaces instead of separate models per surface?
- A. It removes the need for A/B tests
- B. It's always cheaper to serve
- C. Separate models can't use transformers
- D. Each surface alone sees little data; a shared model learns from all of them and passes knowledge on to smaller models
View answer
✓ D. Each surface alone sees little data; a shared model learns from all of them and passes knowledge on to smaller models
Signals on one surface improve predictions on another; knowledge reaches surface models through distillation, representations or shared parameters.
Question 16 of 16
A foundation model is added as one retriever in an ensemble. How should it be trained to add the most value?
- A. Find what the other retrievers miss, treating their predictions as hard negatives
- B. Use only popular items
- C. Train it on the ranker's scores
- D. Copy the other retrievers' outputs
View answer
✓ A. Find what the other retrievers miss, treating their predictions as hard negatives
An ensemble gains most from a member that covers different items, not one that repeats the others.