llama.cpp Flags That Actually Matter

-ngl / --gpu-layers — how much lives on the GPU

The single highest-impact flag. Without it, llama.cpp defaults to running everything on CPU. Setting it to 999 asks for every layer on the GPU — llama.cpp automatically caps it at the model's real layer count, so it's always safe to set this too high rather than too low. If the model doesn't fit fully at 999, that's your cue to either drop the quant or start offloading on purpose rather than letting it spill accidentally (see VRAM spill explained).

--ctx-size — how much context you're paying for

This sets the context window, and it isn't free: every extra token of context grows the KV cache. A model that fits comfortably at 8K context can spill badly at 128K if the KV cache growth wasn't accounted for — this is the single biggest reason a config that "fit" during a quick test starts crawling in a long real conversation.

--cache-type-k / --cache-type-v — quantizing the KV cache

The KV cache doesn't have to stay at full precision. Dropping it from f16 to q8_0 roughly halves its VRAM footprint for a small quality cost, which is often the cheapest way to buy back headroom before touching the model's own quant. Going further (q4_0/q4_1) saves more at a more noticeable cost.

--n-cpu-moe — expert offload for Mixture-of-Experts models

MoE-specific, and the reason a 100B+ model can run on a 24 GB card at all: it keeps attention and the KV cache on the GPU (touched every token) while parking a chosen number of expert feed-forward blocks in system RAM (each touched only when the router selects them). Full mechanics in MoE expert offloading — the short version is that the right value trades VRAM for a small generation-speed cost, and the optimal number is different for every card and every model.

--flash-attn — usually a free win

Reduces the memory and compute cost of the attention calculation itself, typically with no quality cost. Worth enabling by default unless you have a specific reason not to.

--parallel — serving more than one request at a time

Splits (or, with --kv-unified, shares) the KV cache pool across multiple concurrent request slots — relevant if you're running several agent sessions or users against one loaded model, less relevant for a single chat.

FAQ

What's a safe default -ngl value?

999 — llama.cpp caps it at the model's actual layer count, so setting it higher than the model has is harmless and guarantees full GPU offload if your VRAM allows it. If it doesn't fit at 999, that's the signal to either drop to a smaller quant or start offloading deliberately instead.

Does KV cache quantization hurt quality?

A little, and less than most people expect. q8_0 is close enough to f16 that it's usually not perceptible in normal use, while roughly halving KV VRAM cost. q4_0/q4_1 save more memory for a more noticeable quality cost. It's rarely the first place to cut — the model's own quant usually matters more — but it's a cheap lever when you're right at the edge of VRAM.

What if I don't know the right --n-cpu-moe value?

Nobody does on a new card without measuring — the right value depends on your exact VRAM, the model's expert count, and your target context length. TurboLLM's auto-fit runs an actual benchmark sweep across candidate offload splits on your hardware and picks the fastest one that fits, rather than you copying a number tuned for someone else's GPU off a forum post.

Skip the flag-hunting

One command, and auto-fit sets every flag above for your exact hardware. New here? Start with Install & first run and Quantization explained.