AIML Rounds RoadmapMocksJobs

Design an Experimentation Platform

Your company runs a few A/B tests by hand. Design a platform that lets teams run hundreds at once, and trust the results.

The situation

Your company runs a few A/B tests by hand, with analysts writing queries for each one. Results take a week, teams argue about definitions, and a broken test recently shipped. Leadership wants hundreds of tests running at once. The interviewer asks:

Your company runs a few A/B tests by hand. Design a platform that lets teams run hundreds at once, and trust the results.

Ask before designing

  • Scale: say 50 million users a day, 300 tests at once, and 2 billion events a day.
  • Where do changes run? Backend, web, mobile apps? Mobile users may not update their app for weeks.
  • What's the unit? Mostly users, but do you need sessions, devices, or cities?
  • How fresh must results be? Daily for decisions, but guardrail alarms in minutes.
  • Who uses it? If non-specialists read the results, the platform must prevent mistakes by design.
  • Must never happen: a slow or failed platform call breaking the product, or a broken test silently shipped.

The idea

Think of a large hospital running many clinical trials. You need a fair way to assign patients, careful records of who actually took what, one agreed way to measure outcomes, and an independent office that checks each trial before anyone reads the results. Assigning patients is the easy part. Trust takes the rest of the work.

AssignLocal hash,no networkFast and safeLog and joinExposures tometric eventsThe real costStats and trustSRM first,then resultsSafe by default
The hash is easy. Trust is the hard part.

Play

Try to break it. Raise the tests and metrics. Then change the sample ratio cutoff.

False alarms and compute

What does scale do to trust?

metric results a day
–
false wins a day, if nothing works
–
healthy tests wrongly flagged each day
–
exposure rows a day
–

Illustrative. Each result is read once a day, at 5% significance, with no real effects. Exposure rows assume each user reaches half of the running tests, with one row per test per day. That's why real systems deduplicate and aggregate early.

With 300 tests and 50 metrics each, a dashboard shows 15,000 results a day. If nothing worked, about 750 would still look significant. So the platform must lead with a few decision metrics, and label the rest as exploratory. At a 0.01 cutoff, 3 healthy tests get a sample ratio alarm each day. Move it to 0.001, and that falls to 0.3, which is why large platforms use stricter cutoffs. Now look at the exposure rows: they grow with tests times users, which is why this is the part of the pipeline to design first.

Now say why: why must a platform show only a few decision metrics by default?

Architecture

In one breath

Split it into four jobs. Assignment hashes each user locally, with no network call. Exposure logging records who actually reached each change. A metrics pipeline joins exposures to events each day. A stats engine computes results, with trust checks first. The hard parts aren't the hashing, but making results trustworthy by default: automatic sample ratio checks, one agreed metric definition, and stopping rules built in, so a dashboard can't invite peeking.

Four jobs, plus the rules around them

PartWhat it doesKey choice
Config serviceStores tests, splits, targeting and metrics, with review before launchPushed as data files, not code, so tests launch quickly and non-engineers can create them
Assignment SDKComputes hash(user, salt) mod buckets, inside the app or serverLocal, with no network call and no stored table, and a safe default if the config is missing
Exposure logsRecords when a user actually reached the changeLog at the moment of use, not at assignment
Metrics pipelineJoins exposures to events, and builds per-user daily totalsOne central metric definition, so teams don't disagree on what "orders" means
Stats engineComputes effects, intervals, CUPED, the delta method and correctionsStore per-user totals, so results can be recomputed
Trust and decision layerRuns sample ratio checks and guardrails, then shows the resultHide or flag results if trust checks fail

Running many tests

Use layers. Each layer has its own hash, so a user gets one bucket per layer. Tests in different layers overlap, and tests in the same layer are exclusive. Keep a small domain for big changes that touch many layers. The system tracks which settings belong to which layer, and rejects conflicting tests when they're created.

Getting exposure right

If everyone assigned counts, those who never saw the change dilute the effect. So analyse users from their first exposure, in both groups. Control must log too, at the point where treatment would've changed. Otherwise the groups won't be comparable.

Launch flow

  1. Create the test: hypothesis, primary metric, guardrails, sample size and runtime, filled from past data.
  2. Run automatic checks: is there a control, is the traffic free, is the split valid?
  3. Canary on a small share, watching crashes and latency in near real time.
  4. Ramp up to the planned size.
  5. Read results only at planned looks, once trust checks pass.
  6. Decide by the agreed rule, and record it.

The numbers worth saying out loud

All of these are assumptions for the interview.

QuestionWorkingResult
Assignment calls50 million users × 40 checks a day2 billion a day, about 23,000 a second
As a network call23,000 a second, at a 20 ms wait eachToo slow and too fragile, so hash locally
Exposure rows50 million × 300 tests × half reached7.5 billion a day, before deduplication
Per-user totals7.5 billion × 50 metrics375 billion values at worst, so compute only what each test needs
Storage for exposures7.5 billion × 40 bytesAbout 300 GB a day

The hash is cheap. The join between exposures and events is what costs money, so that's what to optimise: partition by day and user, reuse daily totals, and backfill only what changed.

In practice

What the interviewer will push on

  • Hash quality: test that assignment is random and independent across salts, using A/A tests on past data. A bad hash correlates tests silently.
  • Ramping: widening a test should keep users in their group. Keep the same hash and extend the bucket range. Analyse carefully, as early users have had longer exposure.
  • Mobile: old app versions can't be updated. Ship the config with a safe default, and log the version, because exposure is delayed and uneven.
  • Logged-out users: device or cookie IDs change, so one person can land in both groups. Decide what happens at sign-in, and measure how often it occurs.
  • Peeking by design: show daily guardrails freely, but hide the primary decision until a planned look, or use always-valid intervals.
  • Automatic shutdown: a guardrail alarm can stop a test, using sequential thresholds, as long as the rule was agreed beforehand.
  • Interactions: scan pairs that share a surface, with corrections for how many you check. Real ones appear rare, so treat alarms as leads.
  • Failure modes: if the platform fails, the product must carry on in the default experience, and that period must be flagged in the analysis.
  • Institutional memory: store every test, its result and its decision. Past tests supply variance estimates and show which proxy metrics really predict outcomes.

How real teams approached it

  • Layers made it scale: Google moved from a single layer, which couldn't scale, to overlapping layers and domains. Launch layers handle rollouts apart from ordinary tests.
  • Config as data: Google's tests live in data files that are easy to read and pushed more often than code, so non-engineers can create them, with automatic checks first.
  • Tooling matters as much: the same paper says the infrastructure isn't enough alone, and describes sizing tools, a metrics dashboard and education.
  • Namespaces: PlanOut lets each team allocate a share of a namespace, centred on one kind of unit, to separate experiments over time.
  • Interactions are rare, though not never: Microsoft, running hundreds of tests a day per product, found them very rare, which supports overlap as the default.
  • Scale across the industry: 13 organisations at a 2018 summit together reported more than 100,000 experiment treatments in a year.

The answer that sounds right

"The hard part is assigning users at scale"

It sounds right, since it's billions of calls. But a hash is cheap and can run locally. The hard parts are logging true exposure, joining it to events at a reasonable cost, and stopping people from misreading results. A platform that's fast but untrustworthy just helps you ship wrong answers sooner.

Say it

Your company runs a few A/B tests by hand. Design a platform that lets teams run hundreds at once, and trust the results.

Say your answer out loud

This one's a walkthrough, so aim for about 2 minutes. It feels different from reading it, and that's the point.

Tick what you covered

The interviewer pushes further

Follow-up 1 of 3

Why hash locally rather than call an assignment service?

Answer it out loud, then check.

A service adds latency to every check, and becomes a single point of failure. A hash needs only the user ID and the config, gives the same answer everywhere and needs no stored table. The cost is that config must reach every client, so changes aren't instant.

Follow-up 2 of 3

Results show a win, but the sample ratio check failed. What should the platform do?

Answer it out loud, then check.

Flag it, and hide or grey out the result by default, because the groups may no longer be comparable. Then help people diagnose it, by showing the split by browser, version and day. Allow an override only with a recorded reason.

Follow-up 3 of 3

Two tests touch the same settings. How does the platform stop them clashing?

Answer it out loud, then check.

Each setting belongs to a layer, and a user is in at most one test per layer. Two tests on the same setting must then share a layer, so they split traffic and never overlap. The check runs when the test is created, so the conflict is rejected before launch.

I'd first clarify the scale, the platforms, the units, and who reads the results. Then I'd split it into four jobs. Assignment runs in an SDK, hashing user and salt locally, so there's no network call and no stored table, with a safe default if anything fails. Exposure logs record when a user actually reaches the change, in both groups, because counting everyone dilutes the effect. A daily pipeline joins exposures to events, using one central definition per metric, and stores per-user totals. A stats engine computes intervals, with CUPED and the delta method. For many tests at once, I'd use layers: tests in different layers overlap, and tests in the same layer are exclusive. The hard part is trust, so I'd run sample ratio checks on every test and hide results when they fail, let teams pick only a few decision metrics, and build in planned looks or always-valid intervals. With 300 tests and 50 metrics each, about 750 results a day would look significant by chance. So the platform has to make the right behaviour the default, rather than rely on people being careful.

Key terms and sources

Built from published accounts: Google's overlapping experiments paper, which we read in part, the PlanOut paper and Microsoft's interaction study. No single company's design is shown here, and the volumes and lab numbers are our own illustrations. Uses running many tests, trust checks and sequential testing.

End of Experiment design

Ready? Try the Experiment design quiz.

5 questions across this topic, each with the answer and the lesson to revisit.

Take the quiz