AI ENGINEERING · ROUND 3 OF 3
AI system design
One design question, explored through follow-ups. The question: design an assistant that answers employees' questions across 10 million internal documents, respecting who can see what.
Question 1 of 8
Start with requirements. What would you ask, and what would you assume?
Answer out loud first, then open the model answer.
View model answer
I'd ask who the users are and how many, what kinds of questions they ask, which document sources and formats exist, how fresh answers must be, and the targets for latency and cost. The critical requirement is access control: users must only see answers from documents they're allowed to read.
I'd assume around 50,000 employees, a few questions per user per day, answers in a few seconds, documents updated daily, and that every answer must cite its sources.
What the interviewer listens for
- Users, scale, question types, sources
- Freshness, latency and cost targets
- Access control called out early
- States explicit assumptions
Lessons: Design an Enterprise RAG System
Question 2 of 8
Sketch the high-level architecture.
Answer out loud first, then open the model answer.
View model answer
Two paths. Ingestion: connectors pull documents and permissions from each source, parse and chunk them, embed the chunks, and write them to a vector index plus a keyword index, with metadata such as source, date and access list.
Query: authenticate the user, rewrite the question if needed, run hybrid retrieval filtered by the user's permissions, re-rank, then send the top chunks with the question to the LLM, which answers with citations. Around it: a cache, logging and tracing, and an evaluation pipeline.
What the interviewer listens for
- Separate ingestion and query paths
- Hybrid retrieval + re-ranking
- Permission filter inside retrieval
- Citations, tracing and evals around it
Question 3 of 8
How do you enforce who can see what?
Answer out loud first, then open the model answer.
View model answer
Store each chunk with its document's access control list, synced from the source system, and filter on the user's identity and groups during retrieval, so restricted chunks never reach the model. Telling the LLM not to reveal something isn't a control.
Permissions change, so the sync must be frequent and removals urgent. I'd also cache per user or per permission set, never globally, and test with users who should be denied.
What the interviewer listens for
- ACLs stored with chunks, filtered at retrieval
- Never rely on the prompt for security
- Permission sync and fast revocation
- Caching respects permissions
Lessons: Permissions and Poisoning in RAG, Design an Enterprise RAG System
Question 4 of 8
Ten million documents: how does retrieval scale, and what does it cost?
Answer out loud first, then open the model answer.
View model answer
Ten million documents might be a few hundred million chunks. I'd use an approximate nearest-neighbour index such as HNSW, sharded across machines, accepting a little recall loss for speed, and possibly compress vectors to save memory.
Embedding is the big one-off cost and must be re-run when the model changes, so I'd plan re-indexing as a background job. Re-ranking only the top 50 to 100 candidates keeps the expensive step small.
What the interviewer listens for
- Rough sizing of chunks and vectors
- ANN index, sharding, compression
- Re-embedding cost on model change
- Re-rank only a short list
Lessons: Embeddings and Vector Indexes, RAG at Scale: Slow and Wrong
Question 5 of 8
Documents change every day. How do you keep answers fresh?
Answer out loud first, then open the model answer.
View model answer
Incremental ingestion: connectors detect changed or deleted documents, re-chunk and re-embed just those, and delete old chunks, so the index never serves both versions. Deletions and permission changes are processed first.
I'd track index freshness as a metric, show document dates in citations, and prefer newer sources when they conflict.
What the interviewer listens for
- Incremental updates and deletes
- No stale duplicates
- Freshness metric
- Dates and conflict handling
Lessons: Keeping an Index Current
Question 6 of 8
How do you keep latency and cost under control?
Answer out loud first, then open the model answer.
View model answer
Most cost is the LLM call. I'd route simple questions to a smaller model, keep the prompt stable at the start so prefix caching works, cap the context to the best few chunks, and cache answers to frequent questions per permission set.
For latency, I'd stream the answer, run retrieval steps in parallel, and set budgets for each stage, with a timeout fallback.
What the interviewer listens for
- Routing to smaller models
- Prefix and answer caching (permission-aware)
- Fewer, better chunks
- Streaming and per-stage latency budgets
Lessons: Cost and Latency: Routing, Caching and Batching, Prefix and Prompt Caching
Question 7 of 8
How do you know the system gives good answers, before and after launch?
Answer out loud first, then open the model answer.
View model answer
Before launch: an eval set of real employee questions with known source documents, measuring retrieval recall, answer faithfulness and correctness, and permission tests that must always pass. Every change to chunking, prompts or models runs against it.
After launch: traces for every request, thumbs-up and down, escalation rates, and regular review of failures, which are added back to the eval set.
What the interviewer listens for
- Offline eval: retrieval + faithfulness + correctness
- Permission tests as hard gates
- Online signals and traces
- Failures feed the eval set
Lessons: RAG Evals: Retrieval, Faithfulness and Relevance, Tracing and Observability for LLM Apps and Agents
Question 8 of 8
What can go wrong in production, and how do you roll this out safely?
Answer out loud first, then open the model answer.
View model answer
Failure modes: wrong or outdated answers, permission leaks, prompt injection hidden inside documents, provider outages and cost spikes. I'd add guardrails on outputs, treat document text as untrusted, use retries with backoff and a fallback model, and set budget alerts.
Rollout: start with one department and the cleanest sources, monitor quality and leaks closely, then expand. Keep a feedback button and a clear path to a human or the source document.
What the interviewer listens for
- Lists realistic failure modes (leaks, injection, staleness, outages)
- Mitigations for each
- Gradual rollout with monitoring
- Human fallback
Lessons: Guardrails and Prompt Injection, LLM API Reliability: Rate Limits, Retries and Fallbacks