AIML Rounds RoadmapMocksJobs

KV Cache Compression, Eviction and Offloading

Long conversations are filling GPU memory with KV cache. How do you shrink it, drop some of it, or move it?

The situation

Your assistant now keeps whole order histories in context, often 30,000 tokens or more. GPU memory fills with KV cache, and you can only serve a fraction of the users you could before. Someone suggests dropping old tokens from the cache. The interviewer asks:

Long conversations are filling GPU memory with KV cache. How do you shrink it, drop some of it, or move it?

Why it matters

PagedAttention fixed wasted space. But an 8B model with a 32,000-token conversation needs about 4.3 GB of KV cache, and long-running agents and chats keep growing it. At that point you can't fit enough users on a GPU, however well it's packed. This lesson covers what to do next. For model changes that shrink the cache, such as GQA and MLA, see Attention Variants.

The idea

Think of a busy desk covered in notes. You can write smaller, which saves space but is harder to read. You can throw away notes you think you won't need, and hope you're right. Or you can file some in a drawer and fetch them when needed, which takes a moment but loses nothing.

QuantizeFP8 halves it,2 bits moreSmall lossEvictKeep the tokensthat matterCan't come backOffloadCPU memoryor SSDExact, but slower
Choose by whether you can afford any loss.

Play

Try to break it. Raise the context to 1 million tokens. Then try each method, and compare the resume times.

Fit more conversations

Memory against what you lose
KV cache per conversation
–
conversations in 64 GiB
–
what you risk
–
resume from CPU memory
–
resume from SSD
–
resume by recomputing
–

Illustrative arithmetic for the Llama 3 8B shape, at 128 KiB per token in bf16. Reload assumes 64 GB/s from CPU memory and 7 GB/s from SSD. Recompute assumes 2 FLOPs per parameter per token at half of an H100's 989 TFLOP/s, and ignores attention, so long histories take longer. 2-bit ignores the small overhead for scales.

A 32,768-token conversation needs about 4.3 GB in bf16, so only 16 fit in 64 GiB. An FP8 cache fits 32, and a 2-bit cache 128, each with some risk to accuracy. Keeping 20% of tokens fits 80, but whatever was dropped is gone for good. Now look at resuming a paused conversation: reloading from CPU memory takes about 67 ms, from SSD about 0.6 s, and recomputing about 1.1 s of GPU time. That's why offloading pays for agents and long chats: reloading is usually faster, and leaves the GPU free for others. Push to 1 million tokens, and even a single conversation won't fit in bf16.

Now say why: why is reloading usually better than recomputing for long histories?

How it works

In one breath

Paging stops the KV cache from being wasted, but it doesn't make it smaller. At long contexts and many users, the cache itself fills the GPU. You have three choices. Quantize it: FP8 halves it, and methods like KIVI go to 2 bits. Evict parts of it: keep only the tokens that attention relies on most, which saves memory but can lose information for good. Offload it: move idle conversations to CPU memory or SSD and bring them back later, which is exact but costs a transfer. The right choice depends on whether you can afford any loss.

Shrink, drop or move

ApproachSavesLosesExample
Quantize2× at FP8, more at lower bitsA little accuracy, usually small at 8 bitsFP8 KV cache, KIVI at 2 bits
EvictAs much as you chooseDropped tokens are gone for goodH2O, SnapKV, sliding windows with attention sinks
OffloadGPU memory for idle conversationsNothing, but a transfer delay on returnCPU memory and SSD tiers, LMCache, Mooncake

Quantizing the cache

An FP8 cache halves the memory, so roughly twice as many conversations fit. KIVI goes further. It found keys and values behave differently, so it quantizes keys per channel and values per token, at 2 bits. It reported almost the same quality with 2.6× less peak memory including weights, up to 4× larger batches, and 2.35 to 3.47× throughput. Errors reach every later token, so test on long tasks.

Evicting tokens

  • H2O: a few "heavy hitter" tokens get most of the attention. Keeping them, plus recent tokens, kept quality with 20% of the cache, and gave up to 29× throughput over some offloading-based systems.
  • SnapKV: each head attends to consistent parts of the prompt, which can be spotted from the last few prompt tokens. At 16K input it reported 3.6× faster generation and 8.2× better memory efficiency.
  • Sliding windows: keep recent tokens and the first few attention sinks. Memory stays bounded, but older text is forgotten.

The risk with eviction is that a question may need exactly what was dropped. That's fine for summarising a long chat, and bad for "what was the order number I gave you earlier?"

Offloading to cheaper memory

Idle conversations, such as a user who hasn't replied yet, can move to CPU memory or SSD. When they return, you can either reload their cache or recompute it by running prefill again. LMCache, which shares and stores caches outside GPU memory, reported up to 15× throughput with vLLM on multi-round and document workloads. Mooncake, the system behind Kimi, built its whole service around a shared cache across CPU memory and SSD.

Agents change the rules

Engines usually evict from the end of cached sequences, to protect shared prefixes. A 2026 study argues that agents, which return with long histories again and again, should instead keep the shared prefix and the most recent turns, and evict from the start of the rest, because recomputing late tokens is far more expensive. It reported up to 60% lower average time to first token than SGLang with a GPU-only cache, and up to 76% with a two-tier GPU and CPU cache.

The maths

    Simple arithmetic for the reload-or-recompute choice. Recompute also takes GPU time away from other users, which reload doesn’t.

    In practice

    In practice

    • Start with FP8: it's widely supported and usually safe. Test long-context tasks before going lower.
    • Be careful with eviction in products: users expect exact recall of what they said. Prefer offloading, or keep eviction to tasks like summarising.
    • Plan tiers: GPU memory for active conversations, CPU memory for recently idle ones, and SSD for long pauses. SSDs wear out with heavy writes, which a 2026 study addresses.
    • Route to the cache: send a returning conversation back to the machine that holds its cache. See Prefix Caching.
    • Measure on your traffic: watch cache hit rate, reload time and recompute load. Each paper reports on its own workloads.

    The answer that sounds right

    "Drop the old tokens, the model mostly looks at recent ones"

    It sounds right, since attention favours some tokens. But a later question may need exactly what you dropped, like an order number given early on, and evicted tokens can't come back. For products where exact recall matters, quantize or offload instead.

    Say it

    Long conversations are filling GPU memory with KV cache. How do you shrink it, drop some of it, or move it?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Your support bot forgets an order number from early in a long chat after you enabled KV eviction. Why?

    Answer it out loud, then check.

    Eviction keeps tokens that got the most attention so far, and the order number may not have seemed important until the user asked about it. Dropped tokens can't come back. Use quantization or offloading instead, or retrieve key facts explicitly.

    Follow-up 2 of 3

    When is recomputing a cache better than reloading it?

    Answer it out loud, then check.

    For short histories, prefill is quick, and avoids storing and moving the cache at all. For long histories, reloading from CPU memory is usually much faster, and doesn't take GPU time from other users. SSD sits in between, depending on its speed.

    Follow-up 3 of 3

    Does an FP8 KV cache change the model's output?

    Answer it out loud, then check.

    Slightly, as rounding errors enter every later attention step. It's usually small, but test long-context tasks, and compare against bf16 on your own evaluations.

    Paging removes waste, but at long contexts the cache itself fills the GPU, and an 8B model's 32k-token conversation is about 4 GB. I have three options. Quantize it: FP8 halves it with little risk, and KIVI's 2-bit method reported up to 4 times larger batches. Evict tokens: H2O and SnapKV keep the tokens attention relies on, which saves a lot but loses information for good, so I'd avoid it where exact recall matters. Or offload: move idle conversations to CPU memory or SSD and reload them later. Reloading 4 GB from CPU memory takes tens of milliseconds, against about a second of GPU time to recompute. For agents, I'd keep shared prefixes and recent turns, and route returning conversations to the machine holding their cache.

    Key terms and sources

    Why long contexts make the KV cache the bottleneck, quantizing it, evicting unimportant tokens, offloading to CPU memory or SSD, and when to reload or recompute.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Many recent papers are preprints, and vendor results are their own.