MLOps Practice

Answer: training/serving skew or leakage: offline features differ from what the model sees live

1. Compare logged serving features with the training values for the same transactions: missing or stale features online, different encodings or units.
2. Check for leakage: training features built with information after the transaction time (e.g. counts that include later chargebacks).
3. Fix with point-in-time training data, shared feature definitions, and training on logged serving features.

Follow-up: How would you have caught it before launch? Shadow mode comparing live feature distributions and predictions with offline ones.

Answer: hybrid: batch-generate candidates per user, re-rank online with session context

1. Most of the work (candidate generation from long-term preferences) changes slowly: batch it daily or hourly for active users.
2. Re-rank the candidates online at request time using time, location and current session.
3. Stream only the features where freshness matters (recent clicks); fall back to batch results if online ranking fails.

Follow-up: Why not compute everything online? Cost and latency at 30 million users for little gain on slow-changing parts.

Answer: replace p-value alerts with effect-size thresholds on important features, by slice, tied to performance

1. With large samples, KS flags trivial differences; daily alerts become noise.
2. Monitor effect sizes (PSI or distance metrics) on the features that matter most for the model, with thresholds calibrated on history.
3. Alert on prediction drift and performance (once labels arrive) by slice; give each alert an owner and a runbook.

Follow-up: How do you pick 'important' features? Feature importance or ablations, plus business-critical inputs.

Answer: probably not: ~800 requests only detect large regressions; size the test from the smallest regression that matters

1. Compute the sample size for the regression you care about: smaller effects need many more requests (n grows with 1/δ²).
2. 800 requests may detect a big jump but not a subtle one, and 2 hours misses daily cycles.
3. Extend duration or traffic share, cover peak periods, keep guardrails with automatic rollback, and ramp gradually.

Follow-up: What would you check besides the main metric? Latency, errors, slices (new users, cities), and business guardrails.

Answer: only if the run recorded an immutable data snapshot and lineage; code and weights aren't enough

1. The training tables have changed since; without a snapshot ID or time range you can't reconstruct the data.
2. Lineage links: data snapshot → feature definitions → training run → model version → deployment window.
3. Going forward: immutable training snapshots, automated run tracking, a registry with lineage and model cards.

Follow-up: What else would the auditor want? The model card, evaluation by group, and the features used for that specific prediction (from logs).

Answer: no data validation or per-slice comparison against production before promotion

1. Likely cause: a data issue in that city (pipeline bug, label delay, an event) that the new model learned from.
2. Add data validation (schema, ranges, volumes per slice, label completeness) that stops the pipeline on anomalies.
3. Require the candidate to match or beat production overall and on key slices on the same recent test set; then shadow/canary with automatic rollback.
4. Keep the previous model deployable and alert on per-slice performance.

Follow-up: When should a human approve? Large changes in behaviour, high-risk domains, or when gates are borderline.

Answer: an upstream change: the provider's model alias moved to a new version, or the retrieval index/data changed

1. Check traces: model version per request, retrieval index version, tool behaviour, input mix; compare before vs after.
2. Hosted model aliases (e.g. 'latest') can change behaviour; re-indexing can change retrieved documents.
3. Pin model versions and index snapshots, log the full configuration per request, run the golden-set gate before any upstream change, and alert on post-change shifts in escalations and judge scores.

Follow-up: How would you roll back? Switch back to the previous pinned model version or index snapshot via configuration.