[{"data":1,"prerenderedAt":12},["ShallowReactive",2],{"blog-post-how-much-vram-do-you-need-to-run-a-local-llm":3},{"success":4,"post":5},true,{"id":6,"slug":7,"title":8,"content":9,"meta_description":10,"created_at":11,"updated_at":11},"755313a2-9e31-49f9-9b17-f2eb8ff9fade","how-much-vram-do-you-need-to-run-a-local-llm","How much VRAM do you need to run a local LLM?","\u003Cp>There is no single number. The requirement depends on four variables working together: how many parameters the model has, whether it is quantised and to what precision, what context length it processes, and how many requests you handle concurrently. A 7-billion-parameter model quantised to 4-bit fits comfortably on an 8 GB card; the same model at full 32-bit precision needs closer to 28 GB. Add a large context window or serve multiple users at once, and the requirement grows again.\u003C\u002Fp>\n\n\u003Ch2>Start with parameter count\u003C\u002Fh2>\n\n\u003Cp>Model size is usually stated in billions of parameters: a 7B model, 13B, 70B. As a rough starting point, a model at full 32-bit precision needs roughly 4 bytes per parameter. A 7B model is therefore about 28 GB at full precision. A 13B model is about 52 GB. These numbers matter because they set a floor.\u003C\u002Fp>\n\n\u003Cp>But that floor is almost never what you actually pay. Quantisation changes the game.\u003C\u002Fp>\n\n\u003Ch2>Quantisation shrinks the cost\u003C\u002Fh2>\n\n\u003Cp>Quantisation means storing weights in lower precision. At 4 bits per weight instead of 32, the weights occupy an eighth of the space. The same 7B model quantised to 4-bit needs roughly 4 GB of VRAM. A 13B model at 4-bit needs around 8 GB. An 8-bit quantisation sits in the middle.\u003C\u002Fp>\n\n\u003Cp>The tradeoff is accuracy. Lower precision introduces rounding error. For most tasks, the difference is small enough to be unnoticeable. For applications where precision matters (code generation, mathematics, detailed reasoning), full or 8-bit precision may matter. Model cards and community reports on sites like Hugging Face let you see whether a quantised version still performs acceptably for your use case.\u003C\u002Fp>\n\n\u003Ch2>Context length and batch size add to the cost\u003C\u002Fh2>\n\n\u003Cp>VRAM holds not just the weights but also the data in flight: the input tokens you are processing and the output tokens being generated. Longer context windows (how much chat history or document text the model sees) require more VRAM. Serving multiple concurrent requests also multiplies the cost, because each request needs its own space in memory to work.\u003C\u002Fp>\n\n\u003Cp>If you are processing short, single requests one at a time, this overhead is small. If you serve a team of twenty users, or you run batch inference on large documents, context and concurrency become significant.\u003C\u002Fp>\n\n\u003Ch2>How to work out what you need\u003C\u002Fh2>\n\n\u003Cp>Start by identifying the model you want to run and its parameter count. Then check two sources:\u003C\u002Fp>\n\n\u003Cp>\u003Cstrong>The model card.\u003C\u002Fstrong> Hugging Face model cards often include memory requirements, sometimes for multiple quantisations. This is data from actual users who have run the model, not a theoretical estimate.\u003C\u002Fp>\n\n\u003Cp>\u003Cstrong>Ollama's published requirements.\u003C\u002Fstrong> If you are using Ollama to serve the model locally, its model library states the download size for each model at its default quantisation, and its documentation gives general guidance on the memory needed for common model sizes. Treat that as a starting point rather than a precise figure for your workload.\u003C\u002Fp>\n\n\u003Cp>Neither source will be perfectly exact to your setup — your context length, batch size, and system overhead differ from the next person's — but they give you a reliable starting range. Test with your actual workload if the decision is tight.\u003C\u002Fp>\n\n\u003Ch2>If you don't have enough VRAM: CPU and RAM as a fallback\u003C\u002Fh2>\n\n\u003Cp>Modern LLM frameworks can fall back to using system RAM and CPU when VRAM is exhausted. This is not fast. The model moves weights in and out of VRAM, swapping constantly. Inference that would take seconds with adequate VRAM can take minutes with fallback.\u003C\u002Fp>\n\n\u003Cp>This is useful as a last resort — it lets you run a model at all rather than not run it — but not as a solution for production inference. If you are regularly swapping to system RAM, you need either more VRAM, a smaller quantised model, or a shorter context window.\u003C\u002Fp>\n\n\u003Ch2>Multi-GPU and inference frameworks\u003C\u002Fh2>\n\n\u003Cp>If you have multiple GPUs, you can split the model across them. The requirement per GPU drops. Inference frameworks like vLLM handle this automatically. If your VRAM is split across two GPUs, you can run larger models than a single GPU allows. The latency cost depends on the GPU interconnect; modern setups lose little.\u003C\u002Fp>\n\n\u003Cp>None of this changes the total VRAM you need, only how you distribute it.\u003C\u002Fp>\n\n\u003Ch2>Putting it together\u003C\u002Fh2>\n\n\u003Cp>To estimate what you need: find your target model, check its parameter count, look up the VRAM cost at your preferred quantisation (starting with model cards or Ollama), then leave meaningful headroom above it for context and concurrent requests. How much headroom depends entirely on your context length and how many people use it at once, so the only reliable way to size it is to load the model and watch actual usage under a realistic workload. If it fits with room to spare, you are done. If it does not, either choose a smaller model, use a smaller quantisation, limit context length, or add more VRAM.\u003C\u002Fp>\n\n\u003Cp>For a self-hosted deployment using \u003Ca href=\"\u002F\">Nodus Veritatis\u003C\u002Fa> with Ollama, the model runs entirely on your own GPU with no remote overhead. You pay the full VRAM cost of the model you choose, which means this calculation matters: undersizing VRAM leads to slow inference, oversizing is wasted money. Getting it right is worth the few minutes to check.\u003C\u002Fp>","VRAM requirements depend on model size, quantisation, and context length. There's no single answer.","2026-08-06T12:36:21.090354",1786020280954]