AIML Rounds

AI ENGINEERING · ROUND 3 OF 3

AI system design

One design question, explored through follow-ups. The question: design an assistant that answers employees' questions across 10 million internal documents, respecting who can see what.

8 questions · suggested time 45 min
  1. Question 1 of 8

    Start with requirements. What would you ask, and what would you assume?

    Answer out loud first, then open the model answer.

    View model answer

    I'd ask who the users are and how many, what kinds of questions they ask, which document sources and formats exist, how fresh answers must be, and the targets for latency and cost. The critical requirement is access control: users must only see answers from documents they're allowed to read.

    I'd assume around 50,000 employees, a few questions per user per day, answers in a few seconds, documents updated daily, and that every answer must cite its sources.

    What the interviewer listens for

    • Users, scale, question types, sources
    • Freshness, latency and cost targets
    • Access control called out early
    • States explicit assumptions

    Lessons: Design an Enterprise RAG System

  2. Question 2 of 8

    Sketch the high-level architecture.

    Answer out loud first, then open the model answer.

    View model answer

    Two paths. Ingestion: connectors pull documents and permissions from each source, parse and chunk them, embed the chunks, and write them to a vector index plus a keyword index, with metadata such as source, date and access list.

    Query: authenticate the user, rewrite the question if needed, run hybrid retrieval filtered by the user's permissions, re-rank, then send the top chunks with the question to the LLM, which answers with citations. Around it: a cache, logging and tracing, and an evaluation pipeline.

    What the interviewer listens for

    • Separate ingestion and query paths
    • Hybrid retrieval + re-ranking
    • Permission filter inside retrieval
    • Citations, tracing and evals around it

    Lessons: Design an Enterprise RAG System, RAG End to End

  3. Question 3 of 8

    How do you enforce who can see what?

    Answer out loud first, then open the model answer.

    View model answer

    Store each chunk with its document's access control list, synced from the source system, and filter on the user's identity and groups during retrieval, so restricted chunks never reach the model. Telling the LLM not to reveal something isn't a control.

    Permissions change, so the sync must be frequent and removals urgent. I'd also cache per user or per permission set, never globally, and test with users who should be denied.

    What the interviewer listens for

    • ACLs stored with chunks, filtered at retrieval
    • Never rely on the prompt for security
    • Permission sync and fast revocation
    • Caching respects permissions

    Lessons: Permissions and Poisoning in RAG, Design an Enterprise RAG System

  4. Question 4 of 8

    Ten million documents: how does retrieval scale, and what does it cost?

    Answer out loud first, then open the model answer.

    View model answer

    Ten million documents might be a few hundred million chunks. I'd use an approximate nearest-neighbour index such as HNSW, sharded across machines, accepting a little recall loss for speed, and possibly compress vectors to save memory.

    Embedding is the big one-off cost and must be re-run when the model changes, so I'd plan re-indexing as a background job. Re-ranking only the top 50 to 100 candidates keeps the expensive step small.

    What the interviewer listens for

    • Rough sizing of chunks and vectors
    • ANN index, sharding, compression
    • Re-embedding cost on model change
    • Re-rank only a short list

    Lessons: Embeddings and Vector Indexes, RAG at Scale: Slow and Wrong

  5. Question 5 of 8

    Documents change every day. How do you keep answers fresh?

    Answer out loud first, then open the model answer.

    View model answer

    Incremental ingestion: connectors detect changed or deleted documents, re-chunk and re-embed just those, and delete old chunks, so the index never serves both versions. Deletions and permission changes are processed first.

    I'd track index freshness as a metric, show document dates in citations, and prefer newer sources when they conflict.

    What the interviewer listens for

    • Incremental updates and deletes
    • No stale duplicates
    • Freshness metric
    • Dates and conflict handling

    Lessons: Keeping an Index Current

  6. Question 6 of 8

    How do you keep latency and cost under control?

    Answer out loud first, then open the model answer.

    View model answer

    Most cost is the LLM call. I'd route simple questions to a smaller model, keep the prompt stable at the start so prefix caching works, cap the context to the best few chunks, and cache answers to frequent questions per permission set.

    For latency, I'd stream the answer, run retrieval steps in parallel, and set budgets for each stage, with a timeout fallback.

    What the interviewer listens for

    • Routing to smaller models
    • Prefix and answer caching (permission-aware)
    • Fewer, better chunks
    • Streaming and per-stage latency budgets

    Lessons: Cost and Latency: Routing, Caching and Batching, Prefix and Prompt Caching

  7. Question 7 of 8

    How do you know the system gives good answers, before and after launch?

    Answer out loud first, then open the model answer.

    View model answer

    Before launch: an eval set of real employee questions with known source documents, measuring retrieval recall, answer faithfulness and correctness, and permission tests that must always pass. Every change to chunking, prompts or models runs against it.

    After launch: traces for every request, thumbs-up and down, escalation rates, and regular review of failures, which are added back to the eval set.

    What the interviewer listens for

    • Offline eval: retrieval + faithfulness + correctness
    • Permission tests as hard gates
    • Online signals and traces
    • Failures feed the eval set

    Lessons: RAG Evals: Retrieval, Faithfulness and Relevance, Tracing and Observability for LLM Apps and Agents

  8. Question 8 of 8

    What can go wrong in production, and how do you roll this out safely?

    Answer out loud first, then open the model answer.

    View model answer

    Failure modes: wrong or outdated answers, permission leaks, prompt injection hidden inside documents, provider outages and cost spikes. I'd add guardrails on outputs, treat document text as untrusted, use retries with backoff and a fallback model, and set budget alerts.

    Rollout: start with one department and the cleanest sources, monitor quality and leaks closely, then expand. Keep a feedback button and a clear path to a human or the source document.

    What the interviewer listens for

    • Lists realistic failure modes (leaks, injection, staleness, outages)
    • Mitigations for each
    • Gradual rollout with monitoring
    • Human fallback

    Lessons: Guardrails and Prompt Injection, LLM API Reliability: Rate Limits, Retries and Fallbacks