Best local LLMs for the RTX 5070 (12 GB) — updated July 2026
The RTX 5070 pairs Blackwell's 672 GB/s of GDDR7 bandwidth with the same 12 GB that defined
the last generation's mid-range — so it runs the 12 GB shortlist, just faster. In VRAM, that
means dense models up to ~14B at Q4–Q5 and an 8B near-lossless at
Q8_0; with expert offload, a 35B-class MoE like Qwen3.6 35B-A3B
is realistic, because only ~3B params are active per token. What 12 GB honestly can't do,
on any architecture: a usable dense 70B.
Numbers you can trust
TurboLLM measures real tokens/sec on your RTX 5070 — from actual generation, not a
spec sheet — and shows a VRAM-fit verdict before you load. Every speed on this page is
labeled with its source; the only numbers we measured ourselves came from a different card
(an RTX 5070 Ti 16 GB), and everything quoted for the 5070 itself is community-reported.
How this shortlist was built — the 12 GB blend
Our models hub tiers its picks at 8 GB and 16 GB — there's no 12 GB
tier, so this page blends the two, the same way our RTX 4070 Ti
guide does: the 8 GB picks promoted to the higher quants that now fit
fully (Q6/Q8, checked against the actual GGUF file sizes on Hugging Face), plus the 16 GB
picks a quant lower or with CPU offload.
12 GBRTX 5070 — Blackwell, 12 GB GDDR7 at 672 GB/s, between the hub's 8 GB and 16 GB tiers
Small dense models at near-lossless quants, one 35B-class MoE via expert offload, and one dense quality pick that trades speed for it. Sizes are the on-disk GGUF at that quant.
Qwen3.6 35B-A3B
by Alibaba · GGUF
MoE · ~3B active
toolsthinkingcode
The 16 GB tier's daily driver, borrowed downward: with only ~3B params active per token it takes expert offload in stride, so 12 GB runs it at Q3 with the experts parked in system RAM — slower than fully resident, but still the best big-model experience on this card.
RTX 5070:Q3_K_M (16.6 GB) with experts offloaded to system RAM — attention + KV stay on the GPU · wants ~24 GB+ system RAM · 256K context
The 8 GB tier's all-rounder, promoted two quant steps: on 12 GB it runs near-lossless at Q8_0 with VRAM left over for a long context, and its thinking mode is toggleable per request.
RTX 5070:Q8_0 (8.7 GB) fully in VRAM · or Q6_K (6.7 GB) for maximum context headroom · 128K context
The quality pick, caveat spelled out: Q3_K_M is 13.6 GB, so a few layers spill to the CPU, and dense models pay real speed for that. Choose it when answer quality beats tokens/sec — the MoE above is the better daily driver.
RTX 5070:Q3_K_M (13.6 GB) with a few layers on CPU — costs speed · fits fully only at a Q2-class quant, which we don't recommend · 256K context
The best dense coder that fits whole — and at Q8_0 on this card you get all of its quality. The newer coding models are large MoEs that really want 16 GB+.
RTX 5070:Q8_0 (8.1 GB) fully in VRAM · or Q6_K (6.3 GB) · 32K context
The vision pick — strong OCR and spatial reasoning, and small enough that even the near-lossless Q8_0 plus its vision projector leaves room for images and context.
RTX 5070:Q8_0 (8.7 GB) + mmproj (~0.8 GB) in VRAM · or Q4_K_M (5.0 GB) for far more image/context headroom · 256K context
We haven't benchmarked an RTX 5070 ourselves, so the table below is what other people's cards report — useful as a ballpark, not a promise:
Model · quant
Generation
Prompt processing
Llama 3.2 1B · Q4_K_M
~101 t/s
~8,300 t/s
Llama 3.1 8B · Q4_K_M
~56 t/s
~2,900 t/s
Qwen2.5 14B · Q4_K_M
~21 t/s
~1,260 t/s
community-reported, localscore.ai's RTX 5070 results, 2025–2026 — averages across user submissions on this card.
Honest footnotes, because these numbers deserve them. A second community source reads much
faster: llama.cpp's own CUDA benchmark thread on GitHub reports Llama 2 7B at
Q4_0 around ~128 t/s generation (~5,200 t/s prompt processing) on an
RTX 5070 in llama-bench's short-context test — both sources are real, and they
differ because context length, KV cache and flags differ. The 14B row also looks low for a
672 GB/s card (a 14B Q4_K_M fits in 12 GB); small submission pools do that.
And the Q8_0 picks above read twice the bytes per token of these Q4 rows, so
expect them somewhat slower. The only numbers we've measured ourselves come from a different
card: on an RTX 5070 Ti 16 GB, Qwen3.6-35B-A3B at a 200K context ran at
74.7 t/s (q4_0 KV) in TurboLLM — that card has more VRAM and offloads fewer
experts, so don't map its number onto the 5070. When you load any of these models, TurboLLM
measures the real tokens/sec on your card and shows it in the model list — that figure beats
anything on this page.
How TurboLLM auto-fits the RTX 5070
Before you load a model, TurboLLM checks it against your card's real free VRAM — including
KV-cache growth at your chosen context — and shows a fit verdict: fits fully,
fits with offload, or won't fit. On load, auto-tune benchmarks
generation on your GPU and stores the measured tokens/sec in the model list.
Auto-fit then sets the GPU-layer count for dense models and the expert-offload split for MoE
models automatically — it's what lands the Q3 35B-A3B above correctly on the first try instead
of after an hour of flag guessing.
FAQ
Can the RTX 5070 run a 70B model?
Not usably. A 70B GGUF at Q4_K_M is about 43 GB — over three times this card's
12 GB — so most of the model would sit in system RAM, and a dense model offloaded that heavily
crawls at a few tokens per second. The smarter big-model path is a Mixture-of-Experts like
Qwen3.6 35B-A3B: only ~3B parameters are active per token, so it stays
responsive with its experts in system RAM.
What quant should I use on 12 GB of VRAM?
For 7–8B models, take Q6_K or Q8_0 — near-lossless, and they fit
fully in 12 GB with room for context. For 13–14B dense models, Q4_K_M or
Q5_K_M. Anything larger needs a lower quant plus CPU offload, which MoE models
tolerate far better than dense ones. The naming is decoded in
Quantization explained.
How fast will Qwen3.6 35B-A3B run on the RTX 5070?
We haven't measured this card, so we won't print a number. On an RTX 5070 Ti 16 GB we measured
74.7 tokens/sec at a 200K context; the 5070 has 12 GB instead of 16, so it offloads more
experts to system RAM and will land lower. TurboLLM benchmarks the model on your own card the
moment you load it and shows the real figure.
Is the RTX 5070 better than the RTX 4070 Ti for local LLMs?
The model fits are identical — both are 12 GB cards, so they run the same shortlist at the
same quants. On paper the 5070 should be faster — 672 GB/s of GDDR7 versus 504 GB/s, and
generation scales with bandwidth — though today's community submission pools are small
enough that the averages on the two cards' pages don't yet cleanly show that gap. Whichever
you own, the models on this page apply — the RTX 4070 Ti guide
works through the same fits.
Should I get the RTX 5070 Ti instead for local LLMs?
If local LLMs are a priority, the extra 4 GB matters more than the extra speed. 16 GB moves
you a full tier up: a 27B dense fits in VRAM at IQ4_XS, and it runs the 35B-A3B
at Q3 with far less expert offload — mostly resident instead of mostly in system RAM. See the
RTX 5070 Ti guide for the details.