Ranked most-capable-first, from a verified catalog
Model sizes and architecture are verified against each model's Hugging Face repo. The
fit uses TurboLLM's own VRAM-fit maths — weights + KV cache + a compute buffer — but
KV is computed per attention type: full, sliding-window (Gemma), and
hybrid linear (Qwen3.5/3.6, Qwen3-Next), where only a few layers keep
a growing cache, so these run far lighter than their size suggests. Big
MoE models are ranked as runnable because their idle experts sit in
system RAM while the GPU holds the ~3–22 B active path. On Apple Silicon the estimate
assumes you've raised the macOS wired-memory limit. When you load one for real,
TurboLLM re-checks against your exact hardware and measures actual
tokens/sec.
Can I run a local LLM with 8 GB of VRAM?
Yes — 7-8B models at q4 quants (about 5 GB on disk) run fully in 8 GB of VRAM with room for context. Bigger Mixture-of-Experts models can also run by offloading idle experts to system RAM.
How much VRAM do I need to run a 70B model?
Llama-3.3-70B-class dense models need roughly 38-43 GB at q4-class quants, so a 48 GB card runs them fully in VRAM. With less VRAM they need partial CPU offload, which is noticeably slower for dense models.
Does system RAM matter for local LLMs?
For Mixture-of-Experts models, a lot: only a few billion parameters are active per token, so the idle experts can sit in system RAM (llama.cpp's --n-cpu-moe) while the GPU holds the active path — that is how a 120B-class MoE runs on a 24 GB card.
Are these fit results exact?
They're honest estimates from a Hugging-Face-verified catalog: weights + KV cache (computed per attention type — full, sliding-window, or hybrid-linear, which most current Qwen models use) + a compute buffer, with big MoE models counted as runnable via expert-offload to system RAM. Results are ranked most-capable-first. TurboLLM re-checks against your exact hardware before you load and then measures actual tokens/sec — the estimates here pick the shortlist, your hardware settles it.