AI ENGINEERING · ROUND 1 OF 3
LLM fundamentals
The interviewer checks that you understand how LLMs work inside, well enough to reason about cost, quality and failure.
Question 1 of 8
Walk me through what happens inside a decoder-only LLM when it generates the next token.
Answer out loud first, then open the model answer.
View model answer
Text is split into subword tokens, and each token becomes an embedding. The sequence passes through a stack of Transformer blocks. In each block, causal self-attention lets every position look at earlier positions, then a feed-forward layer transforms each position, with residual connections and normalisation around both.
At the last position, the final hidden state is projected onto the vocabulary to give logits. A softmax turns them into probabilities, and a decoding rule such as greedy, temperature or top-p picks the next token. That token is appended and the loop repeats, reusing the cached keys and values so earlier tokens aren't recomputed.
What the interviewer listens for
- Tokens → embeddings → stacked blocks (attention + FFN, residuals, norm)
- Causal mask: only earlier tokens are visible
- Logits → softmax → a decoding rule picks the token
- KV cache avoids recomputing the prefix
Lessons: The Transformer, End to End, Decoding: Temperature, Top-k, Top-p and Min-p, Prefill vs Decode
Question 2 of 8
Why does self-attention divide the scores by √d, and what does attention cost as the context grows?
Answer out loud first, then open the model answer.
View model answer
Each score is a dot product of a query and a key with d dimensions, so its spread grows with d. Without scaling, large scores push the softmax towards one-hot, and gradients become tiny. Dividing by √d keeps the scores' variance near 1.
Every token attends to every earlier token, so compute and the attention matrix grow quadratically with sequence length. At inference the bigger practical cost is the KV cache, which grows linearly with context and limits how many requests fit on a GPU. FlashAttention cuts the memory traffic of the attention step, grouped-query attention shrinks the KV cache, and sliding-window attention caps how far each token looks back.
What the interviewer listens for
- Dot-product variance grows with d; scaling keeps the softmax from saturating
- Quadratic compute in sequence length
- KV cache grows linearly and is the serving bottleneck
- Mentions fixes: FlashAttention, GQA, sliding windows
Lessons: Self-Attention (Q, K, V), Multi-Head Attention, KV Cache Management and PagedAttention
Question 3 of 8
How does a base model become a helpful assistant? Name the main training stages after pretraining.
Answer out loud first, then open the model answer.
View model answer
Pretraining teaches next-token prediction on huge amounts of text, which gives knowledge and fluency but not helpful behaviour. Supervised fine-tuning then trains on example conversations, computing the loss only on the assistant's replies, so the model learns the format and style of answering.
Preference training comes next: RLHF with a reward model and PPO, or simpler methods like DPO that learn directly from pairs of better and worse answers. For reasoning, models are also trained with reinforcement learning on verifiable rewards, for example with GRPO on maths and code. Safety training runs through these stages. The main risk in RL is reward hacking, where the model games the reward, so a KL penalty keeps it close to the SFT model.
What the interviewer listens for
- Pretraining = knowledge; SFT = format and behaviour (prompt tokens masked)
- Preference optimisation: RLHF/PPO or DPO
- RL with verifiable rewards for reasoning
- Risk: reward hacking, controlled with a KL penalty
Lessons: SFT and Instruction Tuning, Reward Models and RLHF (PPO), DPO and Preference Optimisation, Reasoning Models: RLVR, GRPO and Test-Time Compute
Question 4 of 8
Your product needs the model to know your internal policies and to answer in a strict format. Would you use prompting, RAG or fine-tuning?
Answer out loud first, then open the model answer.
View model answer
I'd separate the two needs. Knowledge of policies that change is a retrieval problem, so I'd use RAG: the policies stay in an index that updates without retraining, and answers can cite their source.
Format and tone are behaviour. I'd first try a clear prompt with examples, plus structured output or constrained decoding if the format must parse. Only if that isn't consistent enough at scale, and I have good examples, would I fine-tune, and then for behaviour, not facts.
What the interviewer listens for
- Knowledge gap → RAG; behaviour gap → prompt, then fine-tune
- Prompting first: cheapest and fastest to change
- Structured output / constrained decoding for strict formats
- Fine-tuning is a poor way to add facts that change
Lessons: Prompt vs RAG vs Fine-Tuning: How to Choose, Structured Output and Constrained Decoding
Question 5 of 8
Why do LLMs hallucinate, and how would you reduce it in a product?
Answer out loud first, then open the model answer.
View model answer
The model is trained to produce plausible text, and training and many benchmarks reward a confident guess over saying 'I don't know'. When it lacks the facts, it fills the gap fluently.
In a product I'd ground answers in retrieved sources and ask for citations, let the model say it doesn't know, and check answers against the sources, with an automated faithfulness check or a second pass. I'd measure hallucination rate on an eval set rather than hoping prompts fix it.
What the interviewer listens for
- Training rewards plausible guessing; no built-in sense of 'I don't know'
- Grounding with retrieval and citations
- Allow and reward abstaining
- Measure it with evals (faithfulness)
Lessons: Hallucination: Why LLMs Make Things Up, Grounded Answers: Citations, Conflicts and Saying No, RAG Evals: Retrieval, Faithfulness and Relevance
Question 6 of 8
Explain temperature and top-p. What settings would you use for a support bot versus a brainstorming tool?
Answer out loud first, then open the model answer.
View model answer
Temperature divides the logits before the softmax: low temperature sharpens the distribution towards the top token, high temperature flattens it so less likely tokens appear more. Top-p keeps the smallest set of tokens whose probabilities add up to p and samples only from them.
For a support bot I want consistent, factual answers, so a low temperature, near greedy. For brainstorming I want variety, so a higher temperature with top-p or min-p to cut off the junk tail.
What the interviewer listens for
- Temperature reshapes the distribution
- Top-p truncates to a probability mass
- Low for factual tasks, higher for creative ones
- Bonus: min-p adapts to confidence
Question 7 of 8
Why is LLM decoding slow compared with reading the prompt, and how do serving systems speed it up?
Answer out loud first, then open the model answer.
View model answer
Prefill processes all prompt tokens in parallel, which keeps the GPU's compute busy. Decode makes one token per step, and each step must read all the weights and the KV cache from memory, so it's limited by memory bandwidth.
Serving systems batch many requests together with continuous batching, manage the KV cache in pages so more requests fit, cache repeated prompt prefixes, quantise weights to move fewer bytes, and use speculative decoding to get several tokens per big-model pass.
What the interviewer listens for
- Prefill compute-bound; decode memory-bandwidth-bound
- Continuous batching + PagedAttention
- Prefix caching, quantisation
- Speculative decoding keeps the same output distribution
Lessons: Prefill vs Decode, Continuous Batching and Scheduling, Speculative Decoding
Question 8 of 8
How would you evaluate whether a new model or prompt is better for your product?
Answer out loud first, then open the model answer.
View model answer
I'd build an evaluation set from real user requests, including hard and edge cases, with reference answers or clear grading criteria. I'd score with automatic checks where possible, and an LLM judge validated against human ratings where not, swapping answer order to avoid position bias.
Public benchmarks are only a first filter, since they may not match my task and may be contaminated. Before full rollout I'd run the change on a slice of real traffic and watch product metrics and guardrails such as cost, latency and complaint rate.
What the interviewer listens for
- Task-specific eval set from real traffic
- LLM-as-judge validated against humans; handles position bias
- Benchmarks ≠ your task; contamination
- Online test or staged rollout with guardrails
Lessons: RAG Evals: Retrieval, Faithfulness and Relevance, LLM Judges: Pairwise Preferences and Pass/Fail Checks, Benchmarks and Contamination