Best local LLMs for the RTX 4090 (24 GB) — updated July 2026
The RTX 4090 is the fastest consumer card of its generation for local LLMs. Its 24 GB holds a
30B-class dense model fully in VRAM at q4–q5, a 35B-A3B MoE at q4
with headroom, and with expert offload it runs a 120B-class MoE like gpt-oss-120B —
attention and KV cache on the GPU, experts in system RAM. What it honestly won't do: hold a dense
70B at a sensible quant — that still needs ~40 GB. The fits here are identical to the
RTX 3090's (same 24 GB); what the 4090 adds is speed — ~8% more memory
bandwidth (1,008 vs 936 GB/s) and roughly twice the raw compute, which shows most in prompt processing.
Numbers you can trust
TurboLLM measures real tokens/sec on your card and shows a
VRAM-fit verdict before you load — no guessed speeds in the app, ever.
Every speed on this page is labeled with its source: the only numbers we measured ourselves
came from a different card (an RTX 5070 Ti 16 GB), and everything quoted for the 4090 is
community-reported.
24 GBRTX 4090 — Ada Lovelace, 1,008 GB/s memory bandwidth
The shortlist below is the models hub's 24 GB tier, plus the 16 GB tier's dense pick at the higher quant this card allows, with the fit worked out for this card. Everything a 16 GB card runs also runs here — at a higher quant, longer context, or lighter offload.
Qwen3.6 35B-A3B
by Alibaba · GGUF
MoE · ~3B active
toolsthinkingcode
The daily driver for this card — only ~3B params active per token, so it's dramatically faster than a dense model of similar quality, and on 24 GB it sits fully in VRAM at Q4.
RTX 4090:Q4_K_M (22 GB) fully in VRAM with room for context, or Q5_K_M (26.5 GB) + light expert offload for max quality · 256K context
The dense sibling — slower than the MoE but rock-solid quality. A 16 GB card has to squeeze it to Q3/IQ4; the 4090 runs it at Q4 with ~7 GB left over for a long context.
RTX 4090:Q4_K_M (16.8 GB) fully in VRAM with generous KV-cache headroom · 256K context
The strongest local coding MoE. Its experts live in system RAM while attention and KV stay on the GPU — 24 GB means a higher quant and a lighter offload than a 16 GB card needs.
RTX 4090:IQ3–IQ4_XS (28–38 GB) with CPU-expert offload — auto-fit sets the split · large context
A frontier-class 120B reasoning MoE at native MXFP4. The ~63 GB of weights are mostly experts, so they park in system RAM while attention and KV stay in VRAM — plan on plenty of RAM.
RTX 4090:MXFP4 (~63 GB), experts offloaded to CPU RAM — auto-fit sets it up · 128K context
The vision pick — Google's dense multimodal flagship with the stronger OCR the Qwen picks lack. Fits fully in VRAM at Q4 with room for images and context.
RTX 4090:Q4_K_M (18.3 GB) fully in VRAM, or Q6_K (25.2 GB) + light offload · 256K context
We won't print a made-up number for this card. The only speeds we've measured ourselves come from an
RTX 5070 Ti 16 GB (Qwen3.6-35B-A3B at a 200K context: 74.7 t/s) — a different GPU, so
don't extrapolate from it. What the community has published for the 4090 covers earlier-generation
analogues of the picks above:
Model benchmarked (earlier-gen analogue)
Quant
Community-reported generation speed
Qwen3 30B-A3B (MoE, ~3B active)
q4-class
~190–200 t/s
Qwen3-Coder 30B-A3B (coding MoE)
q4-class
~73–87 t/s
Gemma 2 27B (dense)
q4_K_M
~35–40 t/s
Qwen3 32B (dense)
q4_K_M
~30–45 t/s
Qwen3 8B (dense)
q4_K_M
~110–130 t/s
Community-reported, aggregated from llama.cpp GitHub benchmark discussions (via awesomeagents.ai's home-GPU leaderboard), 2025–2026. Qwen3.6-generation numbers for this card aren't broadly published yet.
The pattern is the useful part: an MoE with ~3B active params generates 4–6× faster than a dense model
of similar total size, which is why Qwen3.6 35B-A3B is the daily-driver pick. For
gpt-oss-120B with expert offload we found no credible community numbers on the 4090 —
speed there depends heavily on your system-RAM bandwidth, so TurboLLM measures it on your machine at
load time rather than guessing. If you're weighing this card against its neighbors, the
RTX 3090 runs the exact same shortlist a step slower, and the
RTX 4070 Ti drops to the 16 GB fits.
How TurboLLM auto-fits the RTX 4090
Load any model and TurboLLM benchmarks it on your 4090, storing the measured tokens/sec
in the model list — live numbers from actual generation, not estimates. Before you load, every quant gets
a VRAM-fit verdict against your real 24 GB, including KV-cache growth at your chosen
context. Auto-fit then picks the GPU-layer count and the MoE expert-offload split for you — it's what
puts gpt-oss-120B's experts in system RAM automatically. Details in
the auto-tune docs.
FAQ
Can the RTX 4090 run a 70B model?
Not comfortably. A dense 70B needs roughly 40 GB at q4_K_M, so on 24 GB you're limited to
aggressive ~2-bit quants or heavy CPU offload, and both cost real quality or speed. A 30B-class model
at q4–q5, or a big MoE like gpt-oss-120B with its experts offloaded to system RAM, gives better results
on this card.
What quant should I use on 24 GB of VRAM?
q4_K_M is the default that delivers the best quality per gigabyte. On 24 GB that means
27B–35B models run fully in VRAM with room for context, and smaller models can step up to
q5_K_M or q6_K for near-lossless quality. TurboLLM shows a VRAM-fit verdict
for each quant before you download — the naming is decoded in
Quantization explained.
How fast will Qwen3.6 35B-A3B run on the RTX 4090?
There are no broad community benchmarks for this exact model on the RTX 4090 yet. Its previous
generation, Qwen3 30B-A3B at a q4-class quant, is community-reported around 190–200 tokens/sec on
this card. TurboLLM measures the real number on your machine the moment you load the model, so you
never have to rely on an estimate.
Do I need to compile anything to run local LLMs on the RTX 4090?
No. TurboLLM detects the card and downloads a prebuilt CUDA engine automatically — llama.cpp or a
community fork — with nothing to compile. Running npx turbollm is the whole setup.