LLM Inference Practice

Answer: 63 blocks (1,008 slots), 8 wasted; the contiguous reservation wastes 1,748 of 2,748 slots (64%)

1. 1,000 tokens ÷ 16 = 62.5 → 63 blocks = 1,008 slots; only the last block is partly empty (8 slots).
2. Contiguous: 700 + 2,048 = 2,748 slots reserved, 1,000 used, 1,748 wasted.
3. That waste is why PagedAttention fits several times more requests in the same memory.

Follow-up: Two samples (n = 2) from the same prompt: how many prompt blocks do they need? Still 44 (the 700-token prompt), shared copy-on-write; only the answers get separate blocks.

Answer: 0.3 + 0.025 × 399 ≈ 10.3 s end to end; for perceived speed, TTFT (and streaming)

1. E2E ≈ TTFT + TPOT × (n − 1) = 0.3 + 9.975 ≈ 10.3 s.
2. With streaming, users start reading after 300 ms; 25 ms per token (40 tokens/s) is faster than reading speed.
3. 'Instant' usually means TTFT: prefix caching, shorter prompts, less queueing, chunked prefill.

Follow-up: When does E2E matter more than TTFT? Non-streamed uses: tool calls, JSON for another service, and agents chaining many calls.

Answer: ≈ 1.95× at small batch; probably not at batch 128 without measuring

1. Expected tokens per target pass: (1 − 0.7⁴) / (1 − 0.7) = 0.7599 / 0.3 ≈ 2.53.
2. Cost per step: γc + 1 = 0.3 + 1 = 1.3. Speedup ≈ 2.53 / 1.3 ≈ 1.95×.
3. At batch 128 decode is closer to compute-bound, so verifying extra tokens isn't free; gains shrink and can turn negative. Measure, and let the engine disable speculation under high load.

Follow-up: How would you raise α? Use a draft from the same family (or an EAGLE head trained on the target's features), lower the temperature, or use prompt-lookup drafts for quoting-heavy tasks.

Answer: about 16 GB ÷ 3.35 TB/s ≈ 4.8 ms ideal (≈ 150 tokens/s at 70% efficiency); batch 32 adds 32 KV caches to every step's reads

1. Each decode step reads all 16 GB of weights: 16 / 3,350 GB/s ≈ 4.8 ms → ~210 tokens/s ideal, ~150 realistic.
2. At batch 32 the weights are read once for all 32 sequences, but each sequence's KV cache (e.g. 2k tokens × 128 KiB ≈ 0.27 GB) is also read: 32 × 0.27 ≈ 8.6 GB extra per step.
3. Step time ≈ (16 + 8.6) GB / bandwidth, so throughput ≈ 32 × (16 / 24.6) ≈ 21× batch 1.

Follow-up: What reduces the KV part? GQA/MLA models, FP8 KV cache, shorter contexts, prefix sharing.

Answer: the change broke prefix caching: the shared prefix now differs per request

1. Prefix caching only matches exactly from the first token; a per-request first line makes every prompt unique, so every request recomputes the whole system prompt.
2. That raises prefill work (TTFT) and API input cost (no cached-token discount); decode is unaffected, so TPOT is unchanged.
3. Fix: move the dynamic line after the static system prompt and tools; confirm with the cache hit rate / cached-token metrics.

Follow-up: How would you catch this before release? Track prefix-cache hit rate in staging load tests and alert on drops after deploys.

Answer: ≈ 38 replicas × 4 GPUs = 152 GPUs; an FP8 KV cache brings it to ≈ 19 × 4 = 76

1. Weights 141 GB → a TP-4 replica (4 × 80 GB, ~288 GB usable at 90%), leaving ~147 GB for KV.
2. KV per 8k conversation: 320 KiB × 8,192 ≈ 2.7 GB → ~54 conversations per replica → 2,000 / 54 ≈ 38 replicas → 152 GPUs.
3. FP8 KV halves the cache (≈ 1.34 GB per conversation) → ~109 per replica → 19 replicas → 76 GPUs. FP8 weights alone would not help as much: the KV cache is the bottleneck.
4. Then validate with a load test against TTFT/TPOT SLOs, add headroom (~20–30%), and use prefix-aware routing.

Follow-up: Most chats are much shorter than 8k. How does that change the plan? Paged KV only uses what's needed, so size on the real context distribution (e.g. p95) and let the scheduler preempt the rare long ones.

Answer: longer contexts multiply KV memory and reads per sequence: smaller batches and slower steps

1. Each long conversation holds up to 4× the KV cache, so fewer sequences fit per replica (smaller batch) → less weight-read sharing → higher cost per token.
2. Every decode step reads each sequence's whole cache, so long sequences slow steps for everyone in the batch → worse TPOT; long prefills also stall decodes without chunking.
3. Fixes: retrieve only relevant order history (RAG) or summarise old turns instead of sending everything; FP8 KV cache; prefix caching for repeated history; route long-context requests to a separate pool; chunked prefill; consider a GQA/MLA-heavy model.

Follow-up: How would you decide between 'long context' and 'retrieve the relevant parts'? Compare answer quality on a labelled set against cost and latency per request; long context is simpler, retrieval is usually far cheaper per query.