AIML Rounds

QUIZ

Recommenders and ranking

16 questions on Recommenders and ranking, from 12 lessons. Pick an answer in your head, then open it.

On the roadmap for: ML Engineering, Data Science.

  1. Question 1 of 16

    How does a recommender pick a few items for a user from millions?

    1. A. Show the most popular items to everyone, refreshed once every hour
    2. B. Score every item in the catalogue with the full model
    3. C. In stages: fast retrieval to hundreds, then a heavier ranker
    4. D. Pick items at random, then learn from clicks
    View answer

    ✓ C. In stages: fast retrieval to hundreds, then a heavier ranker

    Scoring millions of items with a big model per request is too slow; retrieval then ranking keeps it fast.

    Lesson: Design a recommendation system

  2. Question 2 of 16

    Why can interleaving compare two rankers with far fewer users than an A/B test?

    1. A. It uses only offline data, so no live users are needed
    2. B. It skips significance testing to save time
    3. C. It tests one ranker at a time on all of the users
    4. D. Each user sees both rankers mixed; clicks pick a winner
    View answer

    ✓ D. Each user sees both rankers mixed; clicks pick a winner

    Comparing within the same user removes between-user noise, so a preference shows up much sooner.

    Lesson: Test Rankers Faster with Interleaving

  3. Question 3 of 16

    You want to learn which image to show each user for each title. Why use a bandit rather than a fixed A/B test?

    1. A. Bandits need no data to start choosing well
    2. B. It shifts traffic to better images while learning
    3. C. Bandits are always unbiased in the end, unlike A/B tests
    4. D. A/B tests can't compare more than two images
    View answer

    ✓ B. It shifts traffic to better images while learning

    Keep a holdout or a proper test alongside it to prove the personalisation actually helps.

    Lesson: Personalise Images with Bandits

  4. Question 4 of 16

    Why keep the keyword index when adding embedding-based product search?

    1. A. Embeddings blur exact tokens like model numbers; keywords keep filters
    2. B. Keeping both saves storage compared with embeddings alone
    3. C. Keyword search is always more relevant than embeddings
    4. D. Embeddings are too slow to search at this scale
    View answer

    ✓ A. Embeddings blur exact tokens like model numbers; keywords keep filters

    Hybrid retrieval runs both and lets the re-ranker combine them; embeddings help most on long-tail queries.

    Lesson: Design Semantic Product Search

  5. Question 5 of 16

    In a feed ranker, why combine several predicted actions in a value model instead of ranking by click probability?

    1. A. Clicks reward regretted clickbait; weighting actions fixes it
    2. B. It's faster to compute than a single click prediction
    3. C. Click probability can't be predicted at all for brand-new posts with no history
    4. D. Value models don't need any training data to work
    View answer

    ✓ A. Clicks reward regretted clickbait; weighting actions fixes it

    Multi-task predictions plus tunable weights let the product balance engagement types without retraining.

    Lesson: Design Feed Ranking

  6. Question 6 of 16

    You train a click model keeping only 2.5% of non-click examples. What must you do before using its predictions in the auction?

    1. A. Multiply every prediction by 40
    2. B. Recalibrate with q = p / (p + (1 − p) / w)
    3. C. Nothing; the probabilities are already correct
    4. D. Retrain the model on all the data
    View answer

    ✓ B. Recalibrate with q = p / (p + (1 − p) / w)

    Downsampling inflates predicted probabilities; the auction needs real click rates, so convert them back.

    Lesson: Design Ad Click Prediction

  7. Question 7 of 16

    A marketplace adds an embedding per listing to its ranking network. Training NDCG jumps, test NDCG doesn't. Why?

    1. A. The embedding is too small to hold each listing's detail
    2. B. Few bookings per listing, so its embedding memorises noise
    3. C. Embeddings never work for ranking problems
    4. D. The test set must contain different listings
    View answer

    ✓ B. Few bookings per listing, so its embedding memorises noise

    Item embeddings need many interactions per item, as with videos or words; scarce bookings make them overfit.

    Lesson: Design Search Ranking with Deep Learning

  8. Question 8 of 16

    Why does a notification bandit penalise templates a user saw recently?

    1. A. To keep the decision logs small and cheap
    2. B. To save money on sending each notification
    3. C. Templates expire after a few days and must be rewritten
    4. D. Fresh messages work better; repeats get tuned out
    View answer

    ✓ D. Fresh messages work better; repeats get tuned out

    A recency penalty with a half-life, plus softmax exploration, keeps messages varied while still favouring strong templates.

    Lesson: Design Notification Optimisation

  9. Question 9 of 16

    In a short-video feed, why train the ranking model online on streaming actions instead of in a daily batch?

    1. A. It removes the need for retrieval
    2. B. Batch training is impossible at scale
    3. C. Interests and new videos change within minutes; online training picks them up, and beat batch by 14–18% AUC in the paper's live test
    4. D. Online training needs no labels
    View answer

    ✓ C. Interests and new videos change within minutes; online training picks them up, and beat batch by 14–18% AUC in the paper's live test

    Fast-moving feeds reward freshness; the reported live test showed a large AUC gain from training in real time.

    Lesson: Design Short-Video Recommendations

  10. Question 10 of 16

    Why did the short-video system use a collisionless embedding table instead of hashing IDs into a fixed table?

    1. A. Collisionless tables use less memory always
    2. B. Hashing makes different IDs share a row (7.73% of user IDs collided on a public dataset), which blurred them; collisionless tables did better
    3. C. Hashing is slower
    4. D. Hashing can't handle new IDs
    View answer

    ✓ B. Hashing makes different IDs share a row (7.73% of user IDs collided on a public dataset), which blurred them; collisionless tables did better

    Shared rows mix up the signals of different users or videos; the paper kept memory in check by admitting and expiring IDs instead.

    Lesson: Design Short-Video Recommendations

  11. Question 11 of 16

    A job recommender upgrades its embedding model. Downstream models were trained on the old vectors. What keeps them working during the switch?

    1. A. Learn a linear map from new vectors to the old space
    2. B. Average old and new vectors
    3. C. Retrain everything the same day
    4. D. Keep both models forever
    View answer

    ✓ A. Learn a linear map from new vectors to the old space

    A new model produces a different vector space; a learned map lets old consumers read new embeddings until they're retrained.

    Lesson: Design Job Recommendations

  12. Question 12 of 16

    Job matching is two-sided. Which measure reflects that?

    1. A. Number of jobs posted
    2. B. Clicks on job posts only
    3. C. Time on page
    4. D. Rewarding matches where recruiters respond (e.g. accepted messages) and capping how often one job is shown
    View answer

    ✓ D. Rewarding matches where recruiters respond (e.g. accepted messages) and capping how often one job is shown

    A match only works if both sides engage, and flooding a few postings hurts recruiters and job seekers alike.

    Lesson: Design Job Recommendations

  13. Question 13 of 16

    A food-delivery app launches groceries. Most new grocery users have long restaurant histories. What does the cold-start design do?

    1. A. Replace the ranker with an LLM
    2. B. Wait until each user has 20 grocery orders
    3. C. Show random groceries
    4. D. Use an LLM to translate restaurant orders and searches into affinities over the grocery taxonomy, fed to the existing ranker as features
    View answer

    ✓ D. Use an LLM to translate restaurant orders and searches into affinities over the grocery taxonomy, fed to the existing ranker as features

    The LLM adds a signal the ranker was missing while the ranker stays fast and well understood.

    Lesson: Design Recommendations for a New Category

  14. Question 14 of 16

    The LLM sometimes outputs grocery categories that don't exist. What's the fix?

    1. A. Raise the temperature
    2. B. Let the ranker learn to ignore them
    3. C. Require exact taxonomy names, validate outputs and drop anything not in the taxonomy
    4. D. Use a bigger model
    View answer

    ✓ C. Require exact taxonomy names, validate outputs and drop anything not in the taxonomy

    Validating against a closed list turns free text into reliable features.

    Lesson: Design Recommendations for a New Category

  15. Question 15 of 16

    Why train one recommendation foundation model across many surfaces instead of separate models per surface?

    1. A. It removes the need for A/B tests
    2. B. It's always cheaper to serve
    3. C. Separate models can't use transformers
    4. D. Each surface alone sees little data; a shared model learns from all of them and passes knowledge on to smaller models
    View answer

    ✓ D. Each surface alone sees little data; a shared model learns from all of them and passes knowledge on to smaller models

    Signals on one surface improve predictions on another; knowledge reaches surface models through distillation, representations or shared parameters.

    Lesson: Design a Shared Recommendation Foundation Model

  16. Question 16 of 16

    A foundation model is added as one retriever in an ensemble. How should it be trained to add the most value?

    1. A. Find what the other retrievers miss, treating their predictions as hard negatives
    2. B. Use only popular items
    3. C. Train it on the ranker's scores
    4. D. Copy the other retrievers' outputs
    View answer

    ✓ A. Find what the other retrievers miss, treating their predictions as hard negatives

    An ensemble gains most from a member that covers different items, not one that repeats the others.

    Lesson: Design a Shared Recommendation Foundation Model