AIML Rounds

AI ENGINEERING · ROUND 2 OF 3

RAG and agents

The interviewer wants to see you build and debug real LLM applications: retrieval, tools, evaluation and safety.

8 questions · suggested time 45 min
  1. Question 1 of 8

    Design the retrieval part of a RAG system over 10,000 company documents. What choices matter most?

    Answer out loud first, then open the model answer.

    View model answer

    First, parsing and chunking: extract text with structure, keep tables intact, and split on headings or paragraphs into chunks big enough to answer from, with a little overlap. Each chunk keeps metadata such as source, date and permissions.

    Then embed the chunks and store them in a vector index, but also keep a keyword (BM25) index, because embeddings blur exact names and codes. At query time I'd run hybrid search, merge the lists, and re-rank the top candidates with a cross-encoder before passing the best few to the model. I'd measure the retriever on its own with recall@k, on questions where I know which chunk holds the answer.

    What the interviewer listens for

    • Parsing and chunking come first
    • Metadata, including permissions
    • Hybrid search (dense + BM25) and re-ranking
    • Measure recall@k of the retriever

    Lessons: RAG End to End, Parsing and Chunking, Advanced RAG: Hybrid Search, Re-ranking, Query Rewriting and GraphRAG

  2. Question 2 of 8

    Your RAG bot gave a confident, wrong answer. How do you debug it?

    Answer out loud first, then open the model answer.

    View model answer

    I'd open the trace for that request and split the problem in two. First retrieval: was the right chunk retrieved at all, and how high was it ranked? If not, I look at parsing, chunking, the query, or whether hybrid search or a re-ranker is needed.

    If the right chunk was there, it's a generation problem: the model ignored it, mixed it with a conflicting source, or the chunk was lost among too many others. Then I'd tighten grounding: quote sources, handle conflicts, allow 'I don't know'. Finally I add the case to the eval set so it can't come back silently.

    What the interviewer listens for

    • Use the trace; separate retrieval from generation
    • Retrieval fixes: parsing, chunking, hybrid, re-ranking
    • Generation fixes: grounding, conflicts, abstaining
    • Add the case to the regression eval set

    Lessons: RAG: Wrong Answer, Where to Look First, Grounded Answers: Citations, Conflicts and Saying No

  3. Question 3 of 8

    How do you evaluate a RAG system?

    Answer out loud first, then open the model answer.

    View model answer

    I evaluate the two halves separately and together. For retrieval: recall@k and MRR on questions where I know which chunk holds the answer. For generation: faithfulness, meaning every claim is supported by the retrieved context, and answer correctness against a reference.

    I'd build the eval set from real questions, use an LLM judge checked against human labels, and run it on every change to prompts, chunking or models. In production I'd add user feedback and escalation rates.

    What the interviewer listens for

    • Retrieval metrics: recall@k, MRR
    • Faithfulness vs correctness
    • Eval set from real questions; judge validated
    • Run on every change; production signals

    Lessons: RAG Evals: Retrieval, Faithfulness and Relevance, Ranking Metrics

  4. Question 4 of 8

    When would you build an agent instead of a fixed workflow?

    Answer out loud first, then open the model answer.

    View model answer

    If the steps are known in advance, a workflow is cheaper, faster and more reliable, so I'd start there. An agent fits when the path depends on what it finds, such as open-ended research or debugging, and when the tools give clear feedback.

    The catch is that errors compound: at 95% per step, a 20-step task succeeds only about a third of the time. So I'd keep agents' tasks short, add checks against real signals like tests, and give them a way to stop and ask.

    What the interviewer listens for

    • Workflow first when steps are known
    • Agent when the path depends on observations
    • Errors compound over steps (0.95²⁰ ≈ 0.36)
    • Verification and bounded autonomy

    Lessons: What Is an Agent? The ReAct Loop, Planning and Reflection

  5. Question 5 of 8

    Your agent has 60 tools and keeps picking the wrong one. What do you do?

    Answer out loud first, then open the model answer.

    View model answer

    Long tool lists eat context and confuse selection. I'd first clean up: merge overlapping tools, give each a clear name, description and argument schema, and remove the ones that are rarely needed.

    Then I'd load tools on demand: retrieve the few relevant to the request, or group them behind sub-agents or skills. I'd measure tool-call accuracy from traces before and after, so I know the fix worked.

    What the interviewer listens for

    • Fewer, clearer tools with good schemas
    • Load tools on demand (tool retrieval, skills, sub-agents)
    • Measure tool-call accuracy from traces

    Lessons: Tool and Function Calling, Agent Skills and Self-Improving Context, Tool-Call Evals: Selection, Arguments, Trajectories and Effects

  6. Question 6 of 8

    What is MCP, and how is it different from function calling?

    Answer out loud first, then open the model answer.

    View model answer

    Function calling is a model feature: the model emits a structured request to call a tool that the application has described to it. MCP, the Model Context Protocol, is an open protocol for how tools and data are offered: a server exposes tools, resources and prompts, and any MCP-compatible client or agent can use them.

    So function calling is how the model asks, while MCP is how tools get plugged in, once, for many apps. For remote servers it also defines transports and authorisation.

    What the interviewer listens for

    • Function calling = model asks for a tool
    • MCP = standard protocol for servers offering tools to any client
    • Write a tool once, reuse it everywhere
    • Remote servers: transport and auth

    Lessons: MCP: The Model Context Protocol, MCP Under the Hood: Messages, Transports and Authorization

  7. Question 7 of 8

    An agent reads emails and can send messages. How could an attacker make it leak data, and how do you stop it?

    Answer out loud first, then open the model answer.

    View model answer

    This is the lethal trifecta: the agent has private data, reads untrusted content (the email), and can send data out. An attacker hides instructions in an email, the model treats them as a request, and it sends private data to the attacker. Filters alone can't reliably catch every phrasing.

    The fix is architectural: remove at least one leg. Limit what it can send and where, keep untrusted content away from privileged actions, require human approval for outbound messages with sensitive data, and give tools least privilege.

    What the interviewer listens for

    • Names the three legs: private data, untrusted input, exfiltration
    • Prompt injection via content, not the user
    • Filters aren't enough
    • Break a leg: permissions, approvals, isolation

    Lessons: Agent Safety: Permissions, Sandboxing and the Lethal Trifecta, Guardrails and Prompt Injection

  8. Question 8 of 8

    How would you measure whether your agent is reliable enough to launch?

    Answer out loud first, then open the model answer.

    View model answer

    One pass on a benchmark isn't enough. I'd build realistic tasks with checkable outcomes and run each several times, tracking pass^k, the chance it succeeds on all k tries, since customers need consistency.

    I'd also measure at several levels from traces: tool-call accuracy, the trajectory such as steps and loops, task success, cost and latency. In production I'd track success, escalations and user corrections, and review failures regularly.

    What the interviewer listens for

    • Realistic tasks with checkable outcomes
    • Repeat runs; pass^k rather than one pass
    • Metrics at tool, trajectory, task and production level
    • Cost and latency too

    Lessons: Agent Evals: End States, pass^k and Cost, Tool-Call Evals: Selection, Arguments, Trajectories and Effects