AIForum.

VRAM Requirements for Local LLMs: How Much Memory Do You Need?

On this page

THE SHORT ANSWER

At 4-bit quantization, budget roughly 2 GB for a 3B model, 5 GB for a 7–8B model, 8 GB for a 13–14B model, and 40 GB for a 70B model — then add headroom for context and runtime overhead. Weight size is only the starting point: the KV cache grows with conversation length, and a model that exactly fills your VRAM at load time will run out mid-session.
Quick facts

The practical answer is that VRAM demand follows a fairly predictable structure: model weights scale with parameter count and quantization level, and everything else — context, KV cache, and runtime overhead — is added on top. At 4-bit quantization, a 3-billion-parameter model needs around 2 GB, a 7-to-8 billion model around 5 GB, a 13-to-14 billion model around 8 GB, and a 70 billion model around 40 GB. Doubling to 8-bit roughly doubles those figures, and full 16-bit precision roughly quadruples them (Textbook of Linux).

This guide is for anyone sizing a local inference setup — choosing a model for a card you already own, or deciding what to buy. It covers the arithmetic behind the estimates, the parts of memory use that the simple rule hides, and how to pick the largest model that fits comfortably rather than exactly.

The arithmetic behind every VRAM estimate

Weights are the easiest component to predict. Weight memory is approximately (parameters in billions) × (bits per weight) ÷ 8 gigabytes. A 70B model at Q4 therefore stores about 70 × 4 ÷ 8 = 35 GB of weights (llamaperf).

This is why quantization dominates sizing conversations. Moving from 4-bit to 8-bit roughly doubles the weight footprint, and moving to full 16-bit precision roughly quadruples it relative to 4-bit (Textbook of Linux). A 13B model that fits comfortably at 4-bit becomes a poor fit at 16-bit, where it needs on the order of 26 GB.

A second, more conservative formula comes from the Ohio Supercomputer Center, which assumes fp16 and adds an explicit context term: estimated GPU VRAM in GB = 2 × parameters (in billions) + 1 × context length (in thousands). For StableCode with 3 billion parameters and a 16k context, that yields 6 GB of weights plus 16 GB of overhead, or 22 GB total (OSC).

The two approaches are not really in conflict; they serve different purposes. The OSC version is a resource-request estimate for shared clusters, where over-requesting wastes an allocation but under-requesting kills a job. For a desktop card running a quantized model, the weight formula plus a context allowance is much closer to what you will actually observe.

Context and KV cache: the part that grows while you work

Weights are static. The KV cache is not. It grows with the length of the conversation, which means memory use at the end of a long session is higher than at load time. The practical consequence is stated bluntly in the Textbook of Linux guide: a model that exactly fills your VRAM at load time will run out part way through a long session (Textbook of Linux).

The llamaperf tables bake in roughly 2–4 GB of headroom for typical 4K–8K contexts and warn that long contexts of 32K or more, along with a full-precision KV cache, add meaningfully more (llamaperf). If you routinely paste long documents into a chat, or run an agent that accumulates tool output, size for the context you actually use — not the default.

An illustrative example: an 8 GB card

Consider a machine with an 8 GB GPU and a desktop environment that also consumes some VRAM. A 7-to-8B model at 4-bit needs about 5 GB for weights, leaving roughly 3 GB for context, KV cache, and runtime overhead. That is a good fit, and the Textbook of Linux guide names it as exactly that: on an 8 GB card, a 7-to-8B model at 4-bit is a good fit (Textbook of Linux).

The same guide notes that a 13B model at 16-bit is not a fit at that capacity and will fall back to CPU. This is the single most common failure mode reported by people new to local inference: the model loads, generation is dramatically slower than expected, and the cause is partial CPU offloading rather than a bad model.

This scenario is illustrative arithmetic, not a measurement. Your own headroom depends on your runtime, your context setting, and what else is using the GPU.

VRAM by model size and quantization

The table below comes from llamaperf’s requirements guide. Values include weights plus roughly 2–4 GB of headroom for typical context lengths and assume Q4_K_M, described there as the most common quantization (llamaperf).

Model sizeQ4_K_MQ5_K_MQ6_KQ8_0FP16
7B5 GB6 GB7 GB8 GB14 GB
13B8 GB10 GB12 GB14 GB26 GB
30B18 GB22 GB26 GB32 GB60 GB
70B40 GB50 GB58 GB70 GB140 GB
120B (MoE)60 GB75 GB88 GB110 GB240 GB

Two details matter when reading this table. First, the 70B Q4_K_M figure of 40 GB sits above the raw 35 GB weight calculation because of context and KV cache — a reminder that the weight formula is a floor, not a target. Second, MoE (mixture-of-experts) models are listed by total parameters rather than active parameters, which is why a 120B MoE entry appears so large even though only a fraction of the weights are used per token (llamaperf).

What fits on each VRAM tier

Working from the hardware side produces the same information from the opposite direction.

VRAMCommon examplesWhat fits
8 GBRTX 3060 8GB, RTX 4060 8GB, 8 GB Apple Silicon Macs7B at any quant; 13B at Q3 with short context
12 GBRTX 3060 12GB, RTX 4070, 16 GB Apple Silicon Macs13B at Q4; 30B at Q3 with short context
16 GBRTX 4060 Ti 16GB, RTX 4070 Ti Super 16GB, RTX 5060 Ti 16GB13B at Q8; 30B at Q3/Q4
24 GBRTX 3090, RTX 4090, RTX 5090 (32 GB)30B at Q4–Q6; 70B at Q2 with offloading
32 GB+Apple Silicon Pro/Max at 32–64 GB, dual 3090/409030B at Q8; 70B at Q4 (single 32 GB+ card or dual 24 GB)
48 GB+RTX A6000, dual 3090/4090, Apple Silicon Max 48 GB+70B at Q4 with full context; 120B-class models

Source: llamaperf’s VRAM requirements guide (llamaperf).

Where these numbers mislead

Offloading is not the same as fitting. The 24 GB row lists “70B at Q2 with offloading.” That configuration runs, but part of the model executes on the CPU, and generation speed reflects it. Treat offloaded configurations as a way to try a model, not as a way to run one comfortably.

Quantization level is a quality tradeoff, not a free dial. Q4_K_M appears throughout these tables because it is the common default, but the guide that produced them also maintains a separate quantization reference for what the codes mean (llamaperf). Dropping to Q3 or Q2 to squeeze a larger model onto a card is a real option, but you are trading model capability for capacity.

Capacity beats throughput. For local inference, VRAM capacity matters more than raw compute throughput: a slower card with more memory beats a faster card with less (Textbook of Linux). Buying decisions that optimize for clocks rather than memory often produce disappointing local-model performance.

Inference and training are different regimes. The OSC guide estimates training memory at roughly 40× the parameter count in billions, so a 7B model implies a minimum of about 280 GB — beyond a single H100 and requiring distributed training. Activation checkpointing can reduce that at the cost of longer runtime (OSC). None of the inference tables above apply to fine-tuning.

How to size your own setup without guessing

  1. Identify usable VRAM. Note that on a desktop system the display compositor and other applications consume some GPU memory, so the number in the specification is an upper bound rather than a budget.
  2. Pick a quantization first. Q4_K_M is the common starting point. It sets the weight footprint more than any other single choice.
  3. Compute weights. Parameters (billions) × bits ÷ 8. A 13B model at 4-bit is roughly 6.5 GB of raw weights, which rounds up to the 8 GB table entry once context is included.
  4. Reserve context headroom. Decide the context length you actually need. If you run long documents or agent loops, reserve more than the 2–4 GB that typical 4K–8K contexts imply.
  5. Test with a long session, not a short prompt. Memory grows as the conversation does. A model that survives a one-line prompt can still fail twenty turns in.
  6. If you hit an out-of-memory error, change one variable at a time. Reduce context length first, then lower the quantization, then step down in model size. Changing all three at once makes it impossible to learn what your card can actually hold.

Choosing a model for the card you have

If you have 8 GB, plan around 7–8B at 4-bit and accept that 13B means a lower quantization and a short context. At 12 GB, 13B at Q4 becomes comfortable. At 16 GB, 13B at Q8 or 30B at Q3/Q4 are both viable depending on how much you value quality versus size. At 24 GB, 30B at Q4–Q6 is the sweet spot, with 70B at Q2 available only with offloading. Above 32 GB, 70B at Q4 becomes realistic, and dual 24 GB cards open the same door (llamaperf).

The next action is straightforward: pick the largest model and quantization combination that leaves measurable headroom after a long session, not the largest one that merely loads. If you want to compare notes with other practitioners, share your GPU, model, quantization, context length, and observed memory use in the community — concrete configurations are far more useful to the next reader than a general recommendation.

FAQ

Can a 70B model run on a single 24 GB card?

Only in a limited configuration. The llamaperf tier table lists 70B at Q2 with offloading for 24 GB cards, which means part of the model runs on the CPU and generation is substantially slower than a fully resident model.

Does a faster GPU beat a GPU with more VRAM for local LLMs?

No. For this workload, VRAM capacity matters more than raw compute throughput — a slower card with more memory beats a faster card with less, because a model that does not fit falls back to CPU execution.

How much VRAM does fine-tuning or training need compared with inference?

Far more. The Ohio Supercomputer Center estimates training memory at roughly 40× the parameter count in billions, so a 7B model implies a minimum of about 280 GB and requires distributed training. Inference-focused VRAM tables do not apply.

Why does my model run out of memory after a long conversation?

The KV cache grows with context length. A model that exactly fills VRAM at load time will run out part way through a long session, so the practical rule is to choose the largest model that fits comfortably rather than exactly.

References

  1. https://www.textbookoflinux.com/chapter/local-models/how-much-vram.html
  2. https://www.osc.edu/resources/getting_started/howto/howto_estimating_and_profiling_gpu_memory_usage_for_generative_ai
  3. https://llamaperf.com/guides/vram-requirements

Related terms

Quantization
Storing model weights at fewer bits per value — 4-bit, 5-bit, 6-bit, or 8-bit — to reduce memory use, at some cost to output quality.
KV cache
Memory the runtime holds for the attention keys and values of the tokens processed so far. It grows as the conversation or context gets longer.
Q4_K_M
A common 4-bit quantization format, frequently used as the default when sizing a model against available VRAM.
Context window
The maximum number of tokens a model can attend to at once. Larger contexts require more memory for the KV cache.
MoE (mixture of experts)
An architecture where only part of the model is active per token. Memory requirements are usually quoted by total parameters, not active parameters.
CPU offloading
Running part of a model on the CPU when it does not fit in VRAM. The model still works, but generation is much slower.

Leave a Reply

Your email address will not be published. Required fields are marked *