How much VRAM do you need to run an LLM locally?
The number you need is not the model's parameter count and it is not the size of the download. It is three things added together, and the third one is the reason so many first attempts crawl.
By Evdokiia Petrovskaia, Director, Dubir GroupFigures checked
Short version: weights + KV cache + overhead. Weights are parameters × bits-per-weight ÷ 8. KV cache grows with context and depends on architecture. Overhead takes about 10% of a dedicated GPU and about 25% of unified memory. For most people the practical answer is: 16 GB runs useful small models, 24 GB runs the 30B class, and 96 GB or more is where the 120B models start.
Part one: the weights
Straightforward arithmetic. A parameter stored at 4.85 bits, the effective width of Q4_K_M once the embedding and output tensors are counted at higher precision, takes 4.85 bits. Multiply by the parameter count, divide by eight, and you have bytes.
| Model | Q3_K_M | Q4_K_M | Q5_K_S | Q8 |
|---|---|---|---|---|
| Gemma 4 12B | 5.4 GB | 6.8 GB | 7.7 GB | 11.9 GB |
| gpt-oss 20B | 9.5 GB | 11.8 GB | 13.4 GB | 20.7 GB |
| Qwen3.6 27B | 12.2 GB | 15.2 GB | 17.4 GB | 26.7 GB |
| Qwen3 30B-A3B | 13.8 GB | 17.2 GB | 19.6 GB | 30.2 GB |
| gpt-oss 120B | 52.9 GB | 65.9 GB | 75.1 GB | 115.6 GB |
Part two: the KV cache
Every token the model has already seen leaves behind a key and a value vector per attention layer, kept so the next token does not force a recomputation of the whole conversation. The cache grows linearly with context, and its size per token is set by architecture, not parameter count:
2 (keys and values) × layers × KV heads × head dimension × 2 bytes
This is where 2026 architectures diverge sharply. Only full-attention layers contribute; sliding-window and linear-attention layers keep a small constant-size state that does not grow with context. The consequence is counter-intuitive:
| Model | Cache at 8k | Cache at 32k |
|---|---|---|
| Gemma 4 12B | 0.13 GB | 0.5 GB |
| Qwen3.6 27B | 0.50 GB | 2.0 GB |
| Qwen3 30B-A3B | 0.75 GB | 3.0 GB |
| gpt-oss 120B | 0.28 GB | 1.1 GB |
A 117B model needs half the cache of a 27B one. If you size a machine from a rule of thumb rather than the actual architecture, this is the number that embarrasses you.
Part three: the memory you do not get
A dedicated GPU keeps back roughly 10% for the CUDA context, activations and, if a monitor is attached, the desktop. Call it 21.6 GB usable on a 24 GB card.
Unified memory is worse, and this is the most common sizing mistake people make. On Apple Silicon and on the AMD Strix Halo machines the CPU and GPU share one pool, macOS enforces a working-set cap, and the operating system still needs its share. Budget 75%: a 128 GB Mac gives a model about 96 GB. That is still far more than any consumer GPU, which is exactly why Macs run the 120B models that a 5090 cannot touch.
What quantisation actually costs
Fitting a model is a trade against quality, and the trade is much better than people expect until it suddenly is not.
| Level | Bits | Perplexity loss |
|---|---|---|
| Q8 | 8.5 | Indistinguishable |
| Q5_K_S | 5.5 | ~0.05% |
| Q4_K_M | 4.85 | ~0.6% |
| Q3_K_M | 3.89 | ~2-3% |
| Q2_K | 3.0 | ~11% |
Q4_K_M is the default for a reason: two thirds off the memory for a difference nobody detects in conversation. Below 4 bits the curve turns sharply, and Q2 is the tier you use when the alternative is not running the model at all.
Putting it together
Qwen3.6 27B at Q4_K_M with 32k of context: 15.2 GB of weights plus 2.0 GB of cache is 17.2 GB, against 21.6 GB usable on a 24 GB card. It fits, with room for a longer context if you want it.
gpt-oss 120B at Q4_K_M: 65.9 GB of weights plus about 1 GB of cache. No single consumer GPU holds it. A 128 GB Mac, with 96 GB usable, holds it easily.
Rather than doing this by hand for every candidate, put your hardware into the local LLM hardware calculator, which runs the arithmetic above against every model in the catalogue and says why the ones that miss, miss. For choosing between machines, see 4090 vs 5090 vs Mac.
Questions, answered
How much VRAM do I need for a 30B model?
About 17.2 GB for the weights at Q4_K_M, plus KV cache. A 30B mixture-of-experts model with 32k of context adds roughly 3 GB of cache, so budget around 21 GB. That fits a 24 GB card with very little room to spare, and a 32 GB card comfortably.
What is KV cache and why does it use memory?
It is the model's memory of the conversation so far: the key and value vectors for every token already processed, held so they do not have to be recomputed for each new token. It grows linearly with context length, and its size depends on the model's architecture rather than its parameter count, which is why a 117B model can need less cache than a 27B one.
Is Q4_K_M good enough, or should I use Q8?
Q4_K_M is good enough for almost everything. It measures about 0.6% worse perplexity than the full 16-bit weights, which nobody notices in chat, summarisation or retrieval. Q8 is twice the memory for a difference you will not detect. What degrades first at 4 bits is strict output formatting, long chains of arithmetic and rare-language output. If a task fails there, step up to Q5 or Q6 before reaching for a bigger model.
Does a Mac with 32GB of unified memory run 32GB of models?
No. macOS reserves a working-set cap and the operating system still needs its share, so budget about 75% of the total, roughly 24 GB of a 32 GB machine. A dedicated GPU loses less, around 10%, to the CUDA context, activations and the desktop if a monitor is plugged in.
Can I split a model across two GPUs?
Yes, and the memory adds up: two 24 GB cards hold a model that needs 40 GB. Speed does not add up in the same way. For single-stream generation the layers are processed in sequence, so you get the memory of both cards and roughly the bandwidth of one.
Why is my model slower than the calculator says?
Almost always because some layers did not fit on the GPU and spilled to system RAM, which costs most of the speed. Check how many layers were offloaded before blaming the estimate. Long prompts also take time to process before the first token appears, which is normal rather than a fault.
We build the software behind numbers like these
Dubir Group is a technology company in Paphos, Cyprus. Payroll systems, internal tooling, and AI that runs on your own hardware when the data cannot leave.