AIForum.

How to Troubleshoot CUDA Out of Memory Errors in Local AI

On this page

THE SHORT ANSWER

CUDA out of memory in local AI usually means weights, KV cache, and activations together exceed available VRAM. This guide gives a diagnostic order that starts with cheap checks — watching nvidia-smi during load, testing short versus long contexts, checking for other GPU consumers — before concluding that the workload genuinely does not fit.
Quick facts

CUDA out of memory during local AI work usually means one thing: the combined footprint of model weights, KV cache, and activations has exceeded the VRAM available to the process. The remedy is not automatically “buy a bigger GPU” — it is to identify which of those three is responsible, because each has a different fix. This guide is for practitioners running local models through llama.cpp, Transformers, or a comparable runtime who have hit CUDA out of memory or PyTorch’s cuda runtime error(2): out of memory and want a diagnostic order that starts with cheap checks and ends with an honest verdict.

Separate the three memory consumers first

Local inference memory is not a single number. The troubleshooting guide at runlocalai.co frames the problem as model weights plus KV cache plus activations exceeding VRAM, and that framing is the most useful starting point because each component scales differently.

  • Model weights are fixed at load time and determined by parameter count and quantization level.
  • KV cache grows with context length and batch size — it is the reason a model can load cleanly and still fail on a long prompt.
  • Activations are transient working memory, and they are typically much larger during training or fine-tuning than during plain inference.

If you cannot tell which component is growing, you cannot choose the right fix. Changing quantization reduces weights but does nothing for the KV cache. Shortening the context shrinks the KV cache but leaves the weights untouched. Switching to a smaller model does both, at the cost of capability.

Step 1: Watch VRAM while the model loads

The lowest-impact diagnostic is to observe the GPU rather than change anything. Run nvidia-smi in a second terminal while the model loads and watch the memory figure climb. The number it reports when loading stops or fails is the actual ceiling you are working against, not the card’s advertised capacity.

watch -n 1 nvidia-smi

Interpret the result before touching any configuration:

SymptomWorking hypothesisNext check
OOM during load, before any prompt is sentWeights alone exceed available VRAMCompare the model’s on-disk size and quantization against the free VRAM figure
Loads fine, OOM only on long inputsKV cache growthRe-run the same prompt at a 2K context window
OOM instantly on a run that worked yesterdayFragmentation or another process holding VRAMCheck the per-process list in nvidia-smi; restart the runtime
OOM after several runs in one sessionCached allocations or retained tensor referencesClear the cache between runs; release unused tensors

These are hypotheses, not conclusions. Each row narrows the search; none of them proves a cause on its own.

Step 2: If the model never finishes loading

A load-time failure is the simplest case to reason about, because the KV cache has not grown yet. The runlocalai.co guide gives a concrete illustration: attempting to load a 70B Q4 model, roughly 40 GB of weights, onto a 24 GB card will fail regardless of prompt length or configuration.

Available fixes, in order of least disruption:

  1. Drop to a smaller quantization. Moving from Q4_K_M to Q3_K_M reduces the weight footprint; the source describes this as roughly halving the VRAM hit on long contexts. Be aware that quantization level affects output quality, and that this change does not reduce KV cache size.
  2. Switch to a smaller model. A 13B or 8B model at the same quantization will fit where a 70B will not, but expect a capability difference.
  3. Shorten the context window. This helps at the margins but cannot rescue a model whose weights alone exceed the card.

Step 3: If the error only fires on long inputs

This is the signature of KV cache pressure. The runlocalai.co guide describes the diagnostic precisely: run the same prompt at a 2K context window and it works, then run it at 16K and it fails. The model is identical, the weights are identical — only the cache has grown.

Fixes for this class of failure:

  • Lower --ctx-size in llama.cpp, or max_position_embeddings in Transformers, so the runtime does not reserve cache for a context you are not using.
  • Enable flash-attention if your runtime supports it. The same source reports it cuts KV cache by roughly 30% on most models. Treat that figure as a source-reported estimate rather than a guarantee for your specific model and runtime version.
  • Reduce batch size if you are processing multiple requests concurrently, since cache scales with the number of live sequences.

The tradeoff is direct: a shorter context window means the model forgets earlier parts of a long conversation or document. If your workflow depends on long inputs, context reduction is a temporary workaround, not a solution.

Step 4: Rule out fragmentation and stale allocations

If a configuration that worked yesterday now fails immediately at load, the model size is probably not the problem. The runlocalai.co guide identifies VRAM fragmentation from previous runs as a distinct cause, with a recognizable symptom: reloading the model fresh produces an OOM even though nothing about the setup changed.

Diagnostic and fixes, from least to most disruptive:

  1. Restart the runtime process entirely rather than reloading the model in place.
  2. In PyTorch, call torch.cuda.empty_cache() between runs. This releases cached-but-unused memory; it does not free memory that live tensors still reference.
  3. Reboot the machine if the runtime restart is insufficient.
  4. nvidia-smi --gpu-reset is described by the source as usually clearing this condition, but it is only safe when no other process is using the GPU. On a machine running a display server or another inference job, it can disrupt that work.

Step 5: Check what else is holding VRAM

Run nvidia-smi and read the per-process list, not just the total. The runlocalai.co guide notes that a Chrome browser with hardware acceleration enabled can hold 1–2 GB of VRAM, and that other consumers such as OBS or games may hold memory you did not account for. The fix is either to close those processes or, on a multi-GPU machine, to set the AI runtime to a different GPU index so it is not competing for the same card.

If you are training rather than inferring

PyTorch’s own FAQ documents a separate family of OOM causes that apply mainly to training loops. The most common is accumulating autograd history across iterations. In the canonical example, total_loss += loss keeps a differentiable graph alive for every step, because loss carries gradient history. The documented fix is to write total_loss += float(loss) instead, which stores a number rather than a graph.

The same FAQ warns against holding references to tensors you no longer need. A tensor assigned to a local variable is not deallocated until that local goes out of scope, and scopes are often larger than expected — a variable defined inside a loop body may live far longer than the iteration that created it. Using del on temporaries, and avoiding storing differentiable variables as object members, both reduce peak memory.

When the honest answer is more VRAM

After the checks above, some workloads simply do not fit. The runlocalai.co guide states plainly that genuinely insufficient VRAM is “the honest answer 60% of the time” once quantization and context have already been reduced. That is a source-reported estimate, not a measurement you can apply to your own machine.

For hardware, the same guide reports that the cheapest path to 24 GB is a used RTX 3090, cited at roughly $700–1,000 at the time of its May 2026 review. Prices on used hardware move quickly and vary by region, so treat that as a dated reference point rather than a current quote.

Verify that the problem is actually resolved

A successful load is not proof of a fix. Verify against the original failing condition:

  1. Re-run the exact prompt, at the exact context length, that produced the OOM.
  2. Watch nvidia-smi during the run, not just after it. You want to see the peak stay below the ceiling with visible headroom.
  3. Run the workload two or three times in sequence to confirm that memory is released between runs rather than accumulating.
  4. If you changed quantization or context length, evaluate output quality separately — a model that runs is not automatically a model that still performs the task acceptably.

What these checks cannot tell you

The figures and thresholds above come from two sources: a troubleshooting guide reviewed in May 2026 and PyTorch’s own FAQ, last updated in August 2026. Neither can predict your exact VRAM ceiling, because that depends on driver version, runtime version, model architecture, attention implementation, and what else is running on the machine. The flash-attention reduction figure and the quantization guidance are reported estimates, not universal constants.

If the error persists after every step here, the missing diagnostic information is usually the peak VRAM reading at the moment of failure and the exact runtime and model configuration in use. Those two data points are what make a community troubleshooting thread answerable. If you share them — GPU model, VRAM, runtime and version, model and quantization, context length, and the exact error string — other practitioners can compare against their own working setups.

FAQ

Does torch.cuda.empty_cache() fix CUDA out of memory?

Only for one specific cause. It releases cached-but-unused memory left over from previous runs, which helps when fragmentation or retained allocations are the problem. It cannot free memory that live tensors still reference, so it will not rescue a model whose weights genuinely exceed your VRAM.

Will a smaller quantization always solve the error?

No. Quantization reduces the weight footprint, which addresses load-time failures. If the OOM only appears on long inputs, the KV cache is the growing component, and a lower-bit quantization will not shrink it. In that case, reduce context length or enable flash-attention instead.

Is nvidia-smi --gpu-reset safe to run?

Only when no other process is using the GPU. The runlocalai.co guide notes it usually clears fragmentation-related OOM conditions, but on a machine running a display server or another inference job it can disrupt that work. Restarting the runtime or rebooting is the safer first attempt.

Why does the model work at 2K context but fail at 16K?

That pattern points to KV cache growth rather than model size. The weights are identical in both runs; only the cache reserved for processed tokens has expanded. Lower --ctx-size in llama.cpp or max_position_embeddings in Transformers, and test again at the length you actually need.

References

  1. https://www.runlocalai.co/troubleshooting/cuda-out-of-memory
  2. https://docs.pytorch.org/docs/main/notes/faq.html

Related terms

KV cache
Memory reserved during inference to store key and value tensors for previously processed tokens. It grows with context length and batch size, which is why a model can load successfully and still fail on a long prompt.
Quantization (Q4_K_M, Q3_K_M)
A compression scheme that reduces the precision of model weights to lower their memory footprint. Lower-bit quantizations use less VRAM but can reduce output quality, and they do not shrink the KV cache.
VRAM fragmentation
A condition where free GPU memory exists but is split into blocks too small to satisfy a new allocation. It can cause an OOM on a configuration that previously worked, and is usually cleared by restarting the runtime or rebooting.
flash-attention
An attention implementation that reduces memory use compared with standard attention. Support depends on the runtime and GPU architecture; the ~30% KV cache reduction figure is source-reported.
nvidia-smi
NVIDIA's command-line GPU monitoring utility. It reports total and per-process VRAM usage, making it the first tool to reach for when diagnosing an out-of-memory condition.

Leave a Reply

Your email address will not be published. Required fields are marked *