Best local LLMs for the RTX 3090 (24 GB) — updated July 2026
24 GB of VRAM is the single most useful spec in local LLMs, and the used RTX 3090 is still the
cheapest way to get it. Dense models up to ~31B run fully in VRAM at
Q4–Q6; with MoE expert offload the ceiling stretches all the way to
gpt-oss-120B. What it won't do: a dense 70B at reasonable quality — that needs
roughly twice this card's memory.
One honest caveat: this is 2020 Ampere silicon, and a 4090 will beat it on the same model.
The gap is smaller than the spec sheets suggest for generation — decode is mostly
memory-bandwidth-bound, and the 3090's 936 GB/s sits close to the 4090's 1008 GB/s — but
prompt processing leans on compute, where Ada pulls clearly ahead. Expect lower tokens/sec
than a 4090 on every model below, especially on long prompts.
Numbers you can trust
TurboLLM measures real tokens/sec on your card and shows a
VRAM-fit verdict before you load. Every speed on this page is
labeled with its source — community-reported figures are marked as such, and nothing here
is a guess dressed up as a measurement.
The shortlist for 24 GB
Five picks from our models hub's 24 GB tier, with the exact quant and offload that fits this card.
Qwen3.6 35B-A3B
by Alibaba · GGUF
MoE · ~3B active
toolsthinkingcode
The daily driver, and the flagship pick for this card — only ~3B params active per token, so it stays fast on Ampere. On 24 GB it runs fully in VRAM at Q4 instead of the Q3 a 16 GB card needs.
RTX 3090:Q4_K_M (22 GB) fully in VRAM with room for context, or Q5_K_M (26.5 GB) + light expert offload for max quality · 256K context
The strongest local coder. Its idle experts live in system RAM while attention and KV stay on the GPU — 24 GB means a noticeably higher quant and lighter offload than a 16 GB card manages.
RTX 3090:IQ3–IQ4_XS (28–38 GB on disk) with CPU expert offload — needs system RAM to hold the spill · large context
A frontier-class 120B reasoning MoE at native MXFP4. It fits via expert offload, but on this card speed is dominated by your system RAM bandwidth, not the GPU — see the honest numbers below before committing to the download.
RTX 3090:MXFP4 (~63 GB), experts offloaded to CPU RAM, attention + KV in VRAM — auto-fit sets the split · 128K context
We haven't benchmarked a 3090 ourselves — our only founder-measured numbers are from an
RTX 5070 Ti 16 GB, a different card. These are what 3090 owners report, and the spread
within a single model is exactly why TurboLLM measures instead of quoting.
Model
Quant / setup
Community-reported speed
Qwen3.6 35B-A3B
Q3_K_M (16.6 GB file) — fully in VRAM at ~23 GB total with context
~120 t/s
Qwen3.6 35B-A3B
Q5_K_XL (26.6 GB), light offload
~75 t/s at 10K context · ~65 t/s at 120K
Qwen3.6 35B-A3B
Q4_K_M, setup tuned for ~128K context
~33 t/s
gpt-oss 120B
MXFP4, experts in DDR4-3200 system RAM
~1.4–1.8 t/s
All community-reported, not measured by us: huggingface.co model discussions and hardware-corner.net, 2025–2026. The gpt-oss-120B figure is from a DDR4-3200 system — with expert offload, system RAM bandwidth matters more than the GPU, so your number depends heavily on your RAM. Load any of these in TurboLLM and it measures the real figure on your box.
How TurboLLM auto-fits the RTX 3090
On first run TurboLLM detects the 3090, downloads a matching CUDA engine build, and benchmarks
each model as you load it — so the speed in your model list is measured on your card, not copied
from a spec sheet. Before you load, a VRAM-fit verdict tells you whether the quant fits 24 GB;
if it doesn't, auto-fit picks the GPU-layer count and, for MoE
models, the CPU expert-offload split for you. No flags to guess, nothing to compile.
FAQ
Can the RTX 3090 run a 70B?
Not comfortably. Llama 3.3 70B at Q4_K_M is a 43 GB file — nearly double this
card's 24 GB. Your options are an aggressive ~2-bit quant with visible quality loss, or heavy
offload that runs slowly because a dense model touches every weight for every token. A big MoE
like gpt-oss-120B with expert offload is usually the smarter path on this card: only the
active ~5B params hit the GPU per token.
What quant should I use on 24 GB?
Q4_K_M is the default sweet spot. The nice thing about 24 GB is how often you can
go higher: Gemma 4 31B fits at Q4_K_M (18.3 GB) with room to spare and runs
Q6_K (25.2 GB) with light offload, and Qwen3.6 27B at Q4_K_M
(16.8 GB) leaves ~7 GB free for long context. Full decoder ring in
Quantization explained.
How fast will Qwen3.6 35B-A3B run?
Community reports (huggingface.co discussions, 2026) range from ~120 t/s at a Q3
quant fully in VRAM down to ~65–75 t/s at Q5 with light offload, and toward
~33 t/s in setups tuned for maximum context. That 4× spread on one card is the point:
quant, context, and offload dominate. TurboLLM measures the real number on your 3090 the
moment you load it.
Do I need to compile anything?
No. npx turbollm detects the 3090's CUDA setup and downloads a matching prebuilt
engine — nothing to compile. Community forks can be built from source with one click if you
want them, and if you already have GGUF files on disk — from LM Studio or manual
downloads — TurboLLM reads those folders in place without re-downloading.
Is a used RTX 3090 still worth buying in 2026?
For VRAM per dollar, yes — 24 GB decides which models you can run at all, and that matters
more than silicon generation. Generation speed is mostly memory-bandwidth-bound, and the
3090's 936 GB/s sits close to a 4090's 1008 GB/s; where the 4090 clearly wins is prompt
processing and batch work, which lean on its much newer compute. If you process very long
prompts all day, weigh the 4090.
Try it on your 3090
One command. It detects the card, fetches a CUDA engine, and measures real tokens/sec on the first model you load.