Hardware

How much VRAM do you need to run an LLM locally?

The VRAM an LLM needs is three numbers added together: the weights, the KV cache, which grows with your context, and the memory your GPU or operating system keeps for itself. It's the third one that makes so many first attempts crawl.

By Evdokiia Petrovskaia, Director, Dubir GroupFigures checked

Weights are parameters × bits-per-weight ÷ 8. The KV cache grows with context and depends on the architecture, and overhead takes about 10% of a dedicated GPU and about 25% of unified memory. In practice 16 GB runs useful small models, 24 GB runs the 30B class and the 120B models start at 96 GB. The local LLM hardware calculator does this sum for your GPU or Mac against every model in its catalogue.

Part one: the weights

This part is plain arithmetic. Q4_K_M works out to 4.85 bits per parameter once the embedding and output tensors are counted at their higher precision, so multiply the parameter count by 4.85, divide by eight, and you have bytes.

ModelQ3_K_MQ4_K_MQ5_K_SQ8
Gemma 4 12B5.4 GB6.8 GB7.7 GB11.9 GB
gpt-oss 20B9.5 GB11.8 GB13.4 GB20.7 GB
Qwen3.6 27B12.2 GB15.2 GB17.4 GB26.7 GB
Qwen3 30B-A3B13.8 GB17.2 GB19.6 GB30.2 GB
gpt-oss 120B52.9 GB65.9 GB75.1 GB115.6 GB

Part two: the KV cache

Every token the model has already seen leaves behind a key and a value vector per attention layer, kept so the next token doesn't force a recomputation of the whole conversation. The cache grows linearly with context, and its size per token comes from the architecture:

2 (keys and values) × layers × KV heads × head dimension × 2 bytes

This is where 2026 architectures differ a lot. Only full-attention layers count, because sliding-window and linear-attention layers keep a small fixed-size state that doesn't grow with context, so the layer count in the config file tells you almost nothing by itself. Which gives some odd results:

ModelCache at 8kCache at 32k
Gemma 4 12B0.13 GB0.5 GB
Qwen3.6 27B0.50 GB2.0 GB
Qwen3 30B-A3B0.75 GB3.0 GB
gpt-oss 120B0.28 GB1.1 GB

A 117B model needs half the cache of a 27B one, so a rule of thumb based on parameter count gets this number badly wrong.

Part three: the memory you do not get

You never get the whole card. A dedicated GPU keeps back roughly 10% for the CUDA context, activations and, if a monitor is attached, the desktop. Call it 21.6 GB usable on a 24 GB card.

Unified memory loses more, and it's easy to get wrong. On Apple Silicon and on the AMD Strix Halo machines the CPU and GPU share one pool, macOS enforces a working-set cap, and the operating system still needs its share. Budget 75%: a 128 GB Mac gives a model about 96 GB. That's still a lot more than any consumer GPU, which is why Macs run the 120B models a 5090 can't load.

What quantisation costs

Storing a model in fewer bits costs some quality, though less than you'd think until you drop below 4 bits.

LevelBitsPerplexity loss
Q88.5Indistinguishable
Q5_K_S5.5~0.05%
Q4_K_M4.85~0.6%
Q3_K_M3.89~2-3%
Q2_K3.0~11%

Q4_K_M is the usual default because it takes about two thirds off the memory for a difference nobody notices in conversation. Below 4 bits the loss climbs fast, and Q2 only makes sense when the alternative is not running the model at all.

Putting it together

Qwen3.6 27B at Q4_K_M with 32k of context: 15.2 GB of weights plus 2.0 GB of cache is 17.2 GB, against 21.6 GB usable on a 24 GB card. It fits, with room for a longer context if you want it.

gpt-oss 120B at Q4_K_M: 65.9 GB of weights plus about 1 GB of cache. No consumer GPU holds it, while a 128 GB Mac, with 96 GB usable, holds it easily.

You don't have to do this by hand for every candidate. Put your hardware into the local LLM hardware calculator, which runs the arithmetic above against every model in the catalogue and says why the ones that miss, miss. For choosing between machines, see 4090 vs 5090 vs Mac.

Questions, answered

How much VRAM do I need for a 30B model?

About 17.2 GB for the weights at Q4_K_M, plus KV cache. A 30B mixture-of-experts model with 32k of context adds roughly 3 GB of cache, so budget around 21 GB. That fits a 24 GB card with very little room to spare, and a 32 GB card comfortably.

What is KV cache and why does it use memory?

It's the model's memory of the conversation so far: the key and value vectors for every token already processed, kept so they don't have to be recomputed for each new token. It grows linearly with context length, and its size per token is set by the architecture (layers, KV heads, head size), which is why a 117B model can need less cache than a 27B one.

Is Q4_K_M good enough, or should I use Q8?

Q4_K_M is good enough for almost everything. It measures about 0.6% worse perplexity than the full 16-bit weights, which nobody notices in chat, summarisation or retrieval. Q8 is twice the memory for a difference you won't detect. What degrades first at 4 bits is strict output formatting, long chains of arithmetic and rare-language output. If a task fails there, step up to Q5 or Q6 before reaching for a bigger model.

Does a Mac with 32GB of unified memory run 32GB of models?

No. macOS reserves a working-set cap and the operating system still needs its share, so budget about 75% of the total, roughly 24 GB of a 32 GB machine. A dedicated GPU loses less, around 10%, to the CUDA context, activations and the desktop if a monitor is plugged in.

Can I split a model across two GPUs?

Yes, and the memory adds up: two 24 GB cards hold a model that needs 40 GB. Speed does not add up in the same way. For single-stream generation the layers are processed in sequence, so you get the memory of both cards and roughly the bandwidth of one.

Why is my model slower than the calculator says?

Almost always because some layers did not fit on the GPU and spilled to system RAM, which costs most of the speed. Check how many layers were offloaded before blaming the estimate. Long prompts also take a while to process before the first token appears, and that's normal.

Private AI, from the spec to the running box

Dubir Group deploys open models on client hardware from Paphos, Cyprus, for the cases where the data is not allowed to reach a third-party API. We size the machine, install it and keep it answering.