Which LLM can I run locally?
Pick your hardware and what you want the model to do: chat, coding, reading PDFs and passports. The calculator lists the open models that fit, how much context they hold, roughly how fast they generate, and for everything that does not fit, why not.
Your hardware
GeForce
Workstation / data centre
GB10 desktops
A dedicated GPU keeps about 90% for the model; the rest goes to context and activations.
What do you want it to do?
The community default and still the sweet spot: about 0.6% off FP16 perplexity, which nobody notices in chat or summarisation. Strict formatting and long arithmetic degrade first.
How much conversation or document the model holds at once. Every token costs KV-cache memory on top of the weights.
New to this? Quantisation and context, in a minute
A model ships as 16-bit numbers. Quantisation stores them in fewer bits, so the same model takes a half or a quarter of the memory and runs faster too, because generation speed is limited by how fast memory can be read. The cost is accuracy, and it is smaller than most people expect: at Q4_K_M, the default everywhere, the measured loss is under 1% and invisible in normal chat. Down at 3 bits it is a few percent and hard tasks get shaky; at 2 bits the model is visibly dumber. Above Q5 you cannot measure the difference without a test harness.
The rule that matters when shopping: a bigger model at 4-bit almost always beats a smaller one at 8-bit for the same memory.
Context length is the model's working memory: how much of the conversation or document it sees at once. It costs memory separately from the weights (the KV cache), which is why the table below sometimes says a model fits at 8k context but not at 128k. Runtimes such as LM Studio can also quantise the KV cache itself (turn on Flash Attention first): 8-bit cache halves that memory and costs essentially nothing.
Learn more: the loss figures are community measurements on real models, not ours. See Artefact2's KL-divergence tables, the llama.cpp k-quants work that defined these formats, and the LM Studio docs for the Flash Attention and KV-cache settings.
13 of 16 models run on RTX 4090 24GB at Q4_K_M with 8k of context. That device has 24 GB of memory, of which roughly 22 GB is yours to fill.
| Model | Good at | Needs | Speed | Context that fits | Verdict |
|---|---|---|---|---|---|
| Qwen3.6 35B-A3B (MoE)3B active parameters over a 35B pool, plus the same hybrid attention as the 27B. | chat, 24/7 automation, long context | 20 GB | ~333 tok/s | 94k | Runs |
| Qwen3-VL 30B-A3B (MoE) visionThe 30B-A3B recipe with eyes: 262k context swallows a whole contract as page images, and 3.3B active parameters keep it quick. | documents & vision, chat, long context | 18 GB | ~302 tok/s | 44k | Runs |
| Gemma 4 31B vision | chat, coding, documents & vision | 19 GB | ~32 tok/s | 27k | Runs |
| Qwen3 30B-A3B (MoE)A 2025 release that refuses to die: only 3.3B parameters run per token, so it is far faster than its size. | chat, coding, 24/7 automation | 18 GB | ~302 tok/s | 40k | Runs |
| Qwen3.6 27BHybrid attention: 48 of its 64 layers are linear-attention with a fixed-size state, so long context stays cheap. | chat, coding, long context | 16 GB | ~37 tok/s | 102k | Runs |
| Gemma 4 26B-A4B (MoE) vision | chat, documents & vision, 24/7 automation | 15 GB | ~263 tok/s | 177k | Runs |
| gpt-oss 20B (MoE)Ships natively at ~4-bit (MXFP4); higher precisions are theoretical. | chat, coding, 24/7 automation | 12 GB | ~277 tok/s | 128k | Runs |
| Gemma 4 12B vision | chat, documents & vision, drafting | 7.3 GB | ~83 tok/s | 237k | Runs |
| Qwen3-VL 8B visionThe one-model answer to "read this invoice" and "describe this photo": DocVQA 96, OCRBench 90, and it fits an 8GB card at 4-bit. | documents & vision, chat | 6.0 GB | ~115 tok/s | 119k | Runs |
| Gemma 4 E4B vision | chat, drafting, 24/7 automation | 4.6 GB | ~222 tok/s | 128k | Runs |
| Gemma 4 E2B visionThe small end of Gemma 4. Also the standard draft model for speculative decoding in front of its bigger siblings. | drafting, 24/7 automation | 2.9 GB | ~434 tok/s | 128k | Runs |
| DeepSeek-OCR-2 (MoE) visionMIT-licensed OCR built around optical compression: a dense page becomes a handful of visual tokens, which is exactly the bulk-scans and multi-page-PDF case. | documents & vision, 24/7 automation | 2.2 GB | ~1750 tok/s | 8k | Runs |
| PaddleOCR-VL 1.6 visionA 0.9B document-parsing specialist that tops OmniDocBench: tables, seals, skewed scans, 109 languages. Old office hardware is enough for it. | documents & vision, 24/7 automation | 0.6 GB | ~1108 tok/s | 128k | Runs |
| gpt-oss 120B (MoE)Ships natively at ~4-bit (MXFP4); higher precisions are theoretical. | chat, coding | 66 GB | n/a | n/a | Needs 65.9 GB, you have 21.6 |
| DeepSeek V4 Flash (MoE)MIT-licensed frontier weights. Its compressed sparse attention makes real KV use even smaller than our estimate; the weights are what keep it out of single-GPU reach. | chat, coding, long context | 161 GB | n/a | n/a | Needs 160.4 GB, you have 21.6 |
| Qwen3.8 2.4T-A95B (MoE)The first Qwen-Max-class model released as open weights: 95B active out of 2.4T, and only 23 of its 92 layers keep a growing KV cache. Thinking mode cannot be turned off, and no machine on this list holds the weights at any quantisation. | chat, coding, long context | 1356 GB | n/a | n/a | Needs 1355.1 GB, you have 21.6 |
Estimates, not benchmarks. Sizes come from published model shapes and GGUF bits-per-weight; speed assumes generation is memory-bandwidth bound and that a real runtime reaches about 60% of the paper bandwidth. Expect ±15%, and expect better with batching or speculative decoding.
What actually fills your VRAM
The whole calculation is four ideas. Open the ones you care about.
Weights: parameters times bits, divided by eight
A model's weights are just numbers, and the memory they take is arithmetic. An 8B model at 16-bit precision is 8 billion times 2 bytes, about 15GB. Quantise it to 4-bit and the same model is around 4.5GB. The common GGUF formats are not exactly 4 or 8 bits either: Q4_K_M averages close to 4.85 bits per weight because the embedding and output layers are kept at higher precision, which is why our numbers sit slightly above the naive calculation.
KV cache: why context length eats memory
Attention keeps a key and a value vector for every token in the conversation so it does not recompute the past on each new token. That cache is separate from the weights and grows linearly with context:
2 × layers × kv_heads × head_dim × 2 bytes per token
For Qwen3.6 27B that is 64KB per token, and its full 256k window costs 16GB, about as much as the 4-bit weights themselves. This is the part people forget when they load a model that "fits" and then hit an out-of-memory error twenty messages in. The 2026 releases attack exactly this: in Qwen3.6 only 16 of the 64 layers keep a growing cache (the rest use linear attention with a fixed-size state), Gemma 4 gives most layers a short sliding window, and DeepSeek V4 compresses the cache outright. Our estimate counts only the full-attention layers, and quantising the cache to 8-bit halves it again on runtimes that support it.
What quantisation costs you
Going from 16-bit to 4-bit cuts memory by roughly four and costs under 1% in measured perplexity at Q4_K_M. In practice, chat, summarisation and answering from retrieved documents survive it comfortably. Strict JSON, long arithmetic and low-resource languages are where the damage shows up first, and the loss compounds fast below 4 bits. The useful rule: a bigger model at 4-bit almost always beats a smaller one at 8-bit for the same memory. A 32B at Q4 is a better machine than a 14B at Q8.
Speed is bandwidth, not teraflops
Generating one token reads the entire active weight set out of memory once. That makes single-stream generation a memory-bandwidth problem, not a compute problem. Divide the card's bandwidth by the size of the weights and you have the ceiling: 1008 GB/s against a 4.5GB model is around 220 tokens/s in theory, and a real runtime reaches roughly 60% of that. It also explains mixture-of-experts models. Qwen3 30B-A3B stores 30B parameters but only reads 3.3B per token, so it runs at the speed of a small model while taking the memory of a large one.
Prompt processing is the opposite: it is compute-bound and parallel, which is why a long document is ingested quickly on an NVIDIA card and slowly on a laptop. And serving several users at once changes the arithmetic entirely, because a batched engine reads the weights once for the whole batch. Total throughput on a busy server is many times the single-stream number above.
When running it yourself is the right call
Two reasons hold up. The first is data: if customer records, contracts or patient files cannot leave your infrastructure, a hosted API is not an option regardless of price. The second is volume, and that one is arithmetic rather than principle. Steady, predictable token volume amortises a GPU quickly; bursty, low volume never does. There is also a middle path, verifiably private inference inside confidential-computing enclaves, which the breakeven page explains.
Running models locally, answered
Which LLM can I run on 8GB, 12GB or 24GB of VRAM?
Roughly: 8GB runs an 8B-class model like Gemma 4 E4B or Qwen3-VL 8B at 4-bit with short context. 12GB adds Gemma 4 12B. 24GB runs the 26-35B class (Gemma 4 31B, Qwen3.6, Qwen3 30B-A3B) at 4-bit, or a smaller model at 8-bit with long context.
The 96GB and 128GB machines are where gpt-oss 120B fits. Document specialists are far smaller: PaddleOCR-VL parses PDFs in under 2GB.
Set your card in the calculator above and it will list them exactly.
How much VRAM do I need for a 70B model?
About 40GB for the weights at 4-bit, before context: two 24GB cards, a 48GB workstation card, or a Mac with 64GB of unified memory. At 8-bit around 70GB, unquantised 140GB.
Worth knowing before you buy for one: the 2026 open releases mostly skipped the dense-70B shape in favour of mixture-of-experts models like gpt-oss 120B, which take a similar amount of memory but run several times faster.
Does quantisation make the model worse?
A bit, and less than people expect. Q4_K_M, the default, measures under 1% worse perplexity than the full 16-bit weights, and nobody spots that in chat, summarisation or retrieval answers. What degrades first is strict formatting, long chains of arithmetic and rare-language output.
Real damage starts below 4 bits: around 2-3% at Q3, and at Q2 the model is visibly dumber. If a task fails at Q4, try Q5 or Q6 of the same model before jumping to a bigger one.
The full ladder, with the measured loss at each step, is in the quantisation picker above.
Which local model can read PDFs, scans or passports?
A vision model, and this is the class where local quality jumped furthest in the last year. A text-only LLM never sees the page; a document vision model reads the image directly, layout and stamps included.
PaddleOCR-VL 1.6 parses tables and 109 languages in under 2GB, DeepSeek-OCR-2 is built for bulk scans, and Qwen3-VL 8B covers "read this invoice and answer questions about it". Pick "Documents & scans" as the goal in the calculator and the table shows which of them your machine runs.
For passports and IDs in production, test on your own scans first; glare and holograms are what trip the small models.
Can I run a local LLM on a Mac?
Yes: install LM Studio or Ollama, download a model at Q4_K_M, done.
Unified memory is the reason Macs punch above their price for large models: a 128GB Mac holds gpt-oss 120B, which no single consumer GPU can. The trade is bandwidth. An M5 Max moves about 614 GB/s against an RTX 4090's 1008 GB/s, so a dense model generates slower, though the small-active MoE models that dominate 2026 narrow the gap.
Anything from an M1 Pro up is worth putting into the calculator; macOS reserves part of the memory, so budget about 75% of the total.
Why is my model slower than the estimate?
Start with the memory. A browser with fifty tabs holds gigabytes, and a model that spills even a few layers out of GPU memory into system RAM loses most of its speed: check Activity Monitor on macOS or Task Manager on Windows, and LM Studio's model page shows exactly how many layers made it onto the GPU.
Then collect the free wins: turn on Flash Attention, and set up speculative decoding with a small draft model (Gemma 4 E2B in front of a bigger Gemma is the standard pairing), which is a 1.5-2x speedup at identical quality.
And a long prompt is processed before the first token appears, so a slow start on a big document is normal, not a fault.
We put these models into production
Dubir builds private AI: the model runs on your hardware, inside your network, and your data never reaches a third-party API. If buying a box is not the right move, we also set up verifiably private inference in confidential-computing enclaves. Either way, we spec it, deploy it and keep it running.