What "Mixture-of-Experts" actually means
A MoE model splits most of its feed-forward layers into many smaller "expert" sub-networks, plus a small router that picks a handful of them for each individual token. A model described as "120B total, ~5B active" has 120B parameters on disk, but only touches roughly 5B of them to produce any single token — a different, small subset each time, chosen by the router.
Why that enables offloading
Since only a handful of experts are actually read on any given token, parking the rest of them in slower system RAM costs comparatively little — most of them sit untouched most of the time anyway. Contrast that with a dense model, where every parameter is used on every token: offload a dense model the same way and you pay the slow system-RAM cost constantly, because there's no "idle" portion to park.
What stays on the GPU, and what moves to system RAM
Attention layers and the KV cache stay on the GPU — they're touched on every single token, so they need to be fast. The router itself is tiny. The MoE feed-forward "expert" blocks are the part that moves to system RAM, since any individual expert is only read when the router actually selects it for a given token.
gpt-oss-120b ships at native MXFP4, around 63 GB on disk. Keep attention and KV in VRAM, park the idle experts in system RAM, and it runs on a single 24 GB card — see the gpt-oss-120b guide. Qwen3-Coder-Next, an 80B-A3B MoE, measured a flat ~40 t/s at any conversation depth on a 16 GB card once auto-tune found the right offload split — see the full measured numbers.
The trade-off: system RAM, and time-to-first-token
The offloaded experts have to live somewhere — budget system RAM in roughly the same ballpark as the portion you're offloading. The other real cost is at the very start of generation: prefill has to read through the CPU-resident experts before the first token appears, so a long prompt can take noticeably longer to start responding even though generation itself, once running, stays fast. The Qwen3-Coder-Next guide shows this caveat with real measured numbers.
Auto-fit benchmarks different offload splits directly on your card and picks the one that
maximizes speed within your VRAM budget, instead of you hand-tuning
--n-cpu-moe by trial and error — more on the flag itself in
llama.cpp flags that actually matter.
FAQ
Do I need a specific GPU for MoE offloading?
No special hardware feature is required — any GPU running llama.cpp can offload experts to system RAM, since it's a memory-placement decision, not a hardware capability. What matters is having enough system RAM free for the offloaded portion, and PCIe bandwidth reasonable enough that fetching active experts doesn't become the bottleneck.
How much system RAM do I need?
Roughly the size of whatever portion of the model you're offloading, plus normal OS and application overhead. A model with most of its experts on CPU needs system RAM in the same ballpark as the model's total on-disk size; auto-fit's VRAM-fit verdict accounts for this when it picks a split.
Is a MoE model "worse" than a dense model of the same active size?
Not in practice — a well-trained MoE with, say, 3B active parameters typically outperforms a dense 3B model, because it has many more total parameters' worth of specialized knowledge to route between even though only a fraction fire on any given token. The trade-off is disk size and (if offloading) system RAM, not output quality.
Run one yourself
See the models hub for MoE picks by VRAM tier, or What can I run? for picks tailored to your exact hardware.