CUDA out of memory during local AI work usually means one thing: the combined footprint of model weights, KV cache, and activations has exceeded the VRAM available to the process. The remedy is not automatically “buy a bigger GPU” — it is to identify which of those three is responsible, because each has a different fix. This guide is for practitioners running local models through llama.cpp, Transformers, or a comparable runtime who have hit CUDA out of memory or PyTorch’s cuda runtime error(2): out of memory and want a diagnostic order that starts with cheap checks and ends with an honest verdict.
Separate the three memory consumers first
Local inference memory is not a single number. The troubleshooting guide at runlocalai.co frames the problem as model weights plus KV cache plus activations exceeding VRAM, and that framing is the most useful starting point because each component scales differently.
- Model weights are fixed at load time and determined by parameter count and quantization level.
- KV cache grows with context length and batch size — it is the reason a model can load cleanly and still fail on a long prompt.
- Activations are transient working memory, and they are typically much larger during training or fine-tuning than during plain inference.
If you cannot tell which component is growing, you cannot choose the right fix. Changing quantization reduces weights but does nothing for the KV cache. Shortening the context shrinks the KV cache but leaves the weights untouched. Switching to a smaller model does both, at the cost of capability.
Step 1: Watch VRAM while the model loads
The lowest-impact diagnostic is to observe the GPU rather than change anything. Run nvidia-smi in a second terminal while the model loads and watch the memory figure climb. The number it reports when loading stops or fails is the actual ceiling you are working against, not the card’s advertised capacity.
watch -n 1 nvidia-smiInterpret the result before touching any configuration:
| Symptom | Working hypothesis | Next check |
|---|---|---|
| OOM during load, before any prompt is sent | Weights alone exceed available VRAM | Compare the model’s on-disk size and quantization against the free VRAM figure |
| Loads fine, OOM only on long inputs | KV cache growth | Re-run the same prompt at a 2K context window |
| OOM instantly on a run that worked yesterday | Fragmentation or another process holding VRAM | Check the per-process list in nvidia-smi; restart the runtime |
| OOM after several runs in one session | Cached allocations or retained tensor references | Clear the cache between runs; release unused tensors |
These are hypotheses, not conclusions. Each row narrows the search; none of them proves a cause on its own.
Step 2: If the model never finishes loading
A load-time failure is the simplest case to reason about, because the KV cache has not grown yet. The runlocalai.co guide gives a concrete illustration: attempting to load a 70B Q4 model, roughly 40 GB of weights, onto a 24 GB card will fail regardless of prompt length or configuration.
Available fixes, in order of least disruption:
- Drop to a smaller quantization. Moving from Q4_K_M to Q3_K_M reduces the weight footprint; the source describes this as roughly halving the VRAM hit on long contexts. Be aware that quantization level affects output quality, and that this change does not reduce KV cache size.
- Switch to a smaller model. A 13B or 8B model at the same quantization will fit where a 70B will not, but expect a capability difference.
- Shorten the context window. This helps at the margins but cannot rescue a model whose weights alone exceed the card.
Step 3: If the error only fires on long inputs
This is the signature of KV cache pressure. The runlocalai.co guide describes the diagnostic precisely: run the same prompt at a 2K context window and it works, then run it at 16K and it fails. The model is identical, the weights are identical — only the cache has grown.
Fixes for this class of failure:
- Lower
--ctx-sizein llama.cpp, ormax_position_embeddingsin Transformers, so the runtime does not reserve cache for a context you are not using. - Enable
flash-attentionif your runtime supports it. The same source reports it cuts KV cache by roughly 30% on most models. Treat that figure as a source-reported estimate rather than a guarantee for your specific model and runtime version. - Reduce batch size if you are processing multiple requests concurrently, since cache scales with the number of live sequences.
The tradeoff is direct: a shorter context window means the model forgets earlier parts of a long conversation or document. If your workflow depends on long inputs, context reduction is a temporary workaround, not a solution.
Step 4: Rule out fragmentation and stale allocations
If a configuration that worked yesterday now fails immediately at load, the model size is probably not the problem. The runlocalai.co guide identifies VRAM fragmentation from previous runs as a distinct cause, with a recognizable symptom: reloading the model fresh produces an OOM even though nothing about the setup changed.
Diagnostic and fixes, from least to most disruptive:
- Restart the runtime process entirely rather than reloading the model in place.
- In PyTorch, call
torch.cuda.empty_cache()between runs. This releases cached-but-unused memory; it does not free memory that live tensors still reference. - Reboot the machine if the runtime restart is insufficient.
nvidia-smi --gpu-resetis described by the source as usually clearing this condition, but it is only safe when no other process is using the GPU. On a machine running a display server or another inference job, it can disrupt that work.
Step 5: Check what else is holding VRAM
Run nvidia-smi and read the per-process list, not just the total. The runlocalai.co guide notes that a Chrome browser with hardware acceleration enabled can hold 1–2 GB of VRAM, and that other consumers such as OBS or games may hold memory you did not account for. The fix is either to close those processes or, on a multi-GPU machine, to set the AI runtime to a different GPU index so it is not competing for the same card.
If you are training rather than inferring
PyTorch’s own FAQ documents a separate family of OOM causes that apply mainly to training loops. The most common is accumulating autograd history across iterations. In the canonical example, total_loss += loss keeps a differentiable graph alive for every step, because loss carries gradient history. The documented fix is to write total_loss += float(loss) instead, which stores a number rather than a graph.
The same FAQ warns against holding references to tensors you no longer need. A tensor assigned to a local variable is not deallocated until that local goes out of scope, and scopes are often larger than expected — a variable defined inside a loop body may live far longer than the iteration that created it. Using del on temporaries, and avoiding storing differentiable variables as object members, both reduce peak memory.
When the honest answer is more VRAM
After the checks above, some workloads simply do not fit. The runlocalai.co guide states plainly that genuinely insufficient VRAM is “the honest answer 60% of the time” once quantization and context have already been reduced. That is a source-reported estimate, not a measurement you can apply to your own machine.
For hardware, the same guide reports that the cheapest path to 24 GB is a used RTX 3090, cited at roughly $700–1,000 at the time of its May 2026 review. Prices on used hardware move quickly and vary by region, so treat that as a dated reference point rather than a current quote.
Verify that the problem is actually resolved
A successful load is not proof of a fix. Verify against the original failing condition:
- Re-run the exact prompt, at the exact context length, that produced the OOM.
- Watch
nvidia-smiduring the run, not just after it. You want to see the peak stay below the ceiling with visible headroom. - Run the workload two or three times in sequence to confirm that memory is released between runs rather than accumulating.
- If you changed quantization or context length, evaluate output quality separately — a model that runs is not automatically a model that still performs the task acceptably.
What these checks cannot tell you
The figures and thresholds above come from two sources: a troubleshooting guide reviewed in May 2026 and PyTorch’s own FAQ, last updated in August 2026. Neither can predict your exact VRAM ceiling, because that depends on driver version, runtime version, model architecture, attention implementation, and what else is running on the machine. The flash-attention reduction figure and the quantization guidance are reported estimates, not universal constants.
If the error persists after every step here, the missing diagnostic information is usually the peak VRAM reading at the moment of failure and the exact runtime and model configuration in use. Those two data points are what make a community troubleshooting thread answerable. If you share them — GPU model, VRAM, runtime and version, model and quantization, context length, and the exact error string — other practitioners can compare against their own working setups.
FAQ
Does torch.cuda.empty_cache() fix CUDA out of memory?
Only for one specific cause. It releases cached-but-unused memory left over from previous runs, which helps when fragmentation or retained allocations are the problem. It cannot free memory that live tensors still reference, so it will not rescue a model whose weights genuinely exceed your VRAM.
Will a smaller quantization always solve the error?
No. Quantization reduces the weight footprint, which addresses load-time failures. If the OOM only appears on long inputs, the KV cache is the growing component, and a lower-bit quantization will not shrink it. In that case, reduce context length or enable flash-attention instead.
Is nvidia-smi --gpu-reset safe to run?
Only when no other process is using the GPU. The runlocalai.co guide notes it usually clears fragmentation-related OOM conditions, but on a machine running a display server or another inference job it can disrupt that work. Restarting the runtime or rebooting is the safer first attempt.
Why does the model work at 2K context but fail at 16K?
That pattern points to KV cache growth rather than model size. The weights are identical in both runs; only the cache reserved for processed tokens has expanded. Lower --ctx-size in llama.cpp or max_position_embeddings in Transformers, and test again at the length you actually need.