What's actually happening
A model needs VRAM for two things: its own weights, and a KV cache that grows as your conversation gets longer. When the two together stop fitting in your GPU's VRAM, the driver doesn't error out — it quietly offloads the overflow to system RAM, reached over the much slower PCIe bus instead of the GPU's own memory bus. Layers that stay on the GPU keep running at full speed; anything spilled to system RAM runs roughly 5–10× slower. Mix the two and your overall speed collapses toward the slow end, not the average.
Why it's silent
There's no out-of-memory crash to alert you, because technically nothing ran out of memory — the
driver's fallback path exists specifically to keep things running rather than fail loudly. The only
symptom is generation suddenly feeling wrong. On Windows, Task Manager's Performance tab shows a
"Shared GPU memory" graph that lights up the moment this is happening; nvidia-smi (or
your vendor's equivalent) shows the same story from the command line.
The three usual triggers
1. The quant was already a hair too big. A model that "fits" at idle with almost no headroom will spill the instant anything else touches VRAM. 2. The KV cache grew. A long conversation or a big prompt can add gigabytes of KV cache on top of a model that fit fine at the start of the chat. 3. Something else opened VRAM. A browser with hardware acceleration, a game, or a second GPU-heavy app all compete for the same pool.
How to fix it
Leave real headroom below 100% VRAM at idle — not just enough for the weights, but enough for the KV cache to grow into. If you're tight on room, quantizing the KV cache (see the FAQ below) is usually the highest-leverage fix before dropping to a smaller model quant. For a Mixture-of-Experts model, deliberate expert offload (see MoE expert offloading) is a controlled version of the same idea — you choose what moves to system RAM, instead of the driver choosing for you mid-generation.
Auto-fit measures your card's real free VRAM — including how much the KV cache will grow at your chosen context length — and shows a VRAM-fit verdict before you load, not after it's already spilling. See the auto-tune docs and quantization explained for the levers it's balancing.
FAQ
How do I know if VRAM is spilling right now?
On Windows, open Task Manager's Performance tab for your GPU and look at the "Shared GPU memory"
graph — any non-zero use while a model is loaded means spill. On any OS, nvidia-smi
(or the equivalent for your vendor) shows dedicated VRAM usage; if it reads at or above your card's
total while generation feels slow, you're spilling.
Does closing other GPU apps help?
Yes, directly. A browser with hardware acceleration, a second monitor, or a game running in the background all reserve VRAM the desktop needs, which shrinks the room left for your model. If a config fits with room to spare when idle but crawls once you start using your machine normally, this is almost always why.
Is quantizing the KV cache safe?
Generally yes. Dropping the KV cache from f16 to q8_0 roughly halves its
VRAM cost for a small, usually imperceptible quality cost, and is one of the highest-leverage fixes
available before you have to drop the model's own quant. Going further (q4_0/
q4_1) saves more memory at a more noticeable quality cost — worth it when the
alternative is spilling to system RAM.
Try it without the guesswork
TurboLLM shows the VRAM-fit verdict for every quant against your card's real free VRAM before you load — see What can I run? for picks tailored to your hardware, or the flags that actually matter if you'd rather tune it by hand.