Most Ollama performance problems trace back to two settings that pull in opposite directions. Context length controls how much text the model can keep in memory at once; GPU offloading controls how much of the model runs on your graphics card. Raising context consumes memory that could otherwise hold model layers, and moving layers to the GPU consumes VRAM that could otherwise hold a longer context. Tuning is the practice of finding the largest context your workload genuinely needs while keeping as much of the model on the GPU as your hardware allows.
This guide is for technically curious beginners and intermediate practitioners running local models. It covers what each setting does, where to change it, how to verify the result, and where the tradeoffs bite.
The two dials and what each one costs
Context length is the maximum number of tokens the model has access to in memory — prompt, conversation history, retrieved documents, and the response being generated all share that budget. Ollama’s FAQ states a default context window of 4096 tokens, overridable with the OLLAMA_CONTEXT_LENGTH environment variable. The separate context-length documentation describes defaults that scale with available VRAM: under 24 GiB of VRAM, 4k context; 24–48 GiB, 32k context; and 48 GiB or more, 256k context. Those two pages describe defaults at different levels of granularity, so the practical takeaway is to check what your own instance actually allocated rather than assuming a number.
GPU offloading is different. It is not a single on/off switch for the whole model. A community tuning guide describes the --gpu parameter as controlling exactly how many layers are offloaded to the GPU, and it recommends thinking of the model as a multi-story building where you decide how many floors sit on GPU “land” versus CPU “land.” Layers that stay on the CPU are computed in system memory, which is generally much slower than VRAM.
The reason these two dials interact is that both draw on the same finite memory pool. Ollama’s documentation notes plainly that setting a larger context length increases the amount of memory required to run a model. A community estimate table for model weights illustrates the same pressure from the other direction — approximate VRAM needs for Q4, Q5, Q8, and FP16 quantization across 7B to 70B models — and explicitly warns that actual memory usage also depends on context length and batch size. Weight size alone is never the full memory picture.
Changing context length in Ollama
There are four places to set context, and they apply at different scopes.
- Ollama app: move the slider under settings to the desired context length. This is the simplest option if you run the desktop application.
- Server-wide default: set the environment variable when serving Ollama. This applies to every request the server handles unless a request overrides it.
- Per-run session: inside an interactive
ollama runsession, use the/set parametercommand. - Per-request via API: pass
num_ctxin theoptionsobject.
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
/set parameter num_ctx 4096
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Why is the sky blue?",
"options": { "num_ctx": 4096 }
}'Note that cloud models are set to their maximum context length by default, so the tuning described here mainly concerns locally served models.
Choosing a context length for your workload
Bigger is not automatically better. Context is a budget you pay for in memory, and unused context is wasted memory that could have held GPU layers. Ollama’s documentation gives one concrete workload recommendation: tasks that require large context, such as web search, agents, and coding tools, should be set to at least 64000 tokens.
For everything else, work backwards from the actual task. A short chat or a single-document summarization job rarely needs more than a few thousand tokens. A retrieval-augmented workflow that stuffs several documents into the prompt will need more, and the amount depends on how many documents you inject and how long they are. Measure your typical prompt size, add headroom for the response, and set context near that number rather than at the maximum your hardware can theoretically reach.
Tuning GPU offloading step by step
The goal is to use as much of the GPU as possible without running out of VRAM. The following procedure follows the community tuning guide’s advice and can be run on any supported platform.
- Identify your GPU and available VRAM. Use
nvidia-smifor NVIDIA cards,rocm-smifor AMD, orsystem_profiler SPDisplaysDataTypeon Apple Silicon. - Start conservative. The guide suggests beginning with roughly half the layers for a model that clearly exceeds your VRAM, for example
ollama run llama3:8b --gpu 16for a 7B model on a 6 GB card. - Increase the layer count gradually. Raise the
--gpuvalue until you either reach the model’s total layer count or hit an out-of-memory error. - Step back from the ceiling. When you encounter an OOM error, reduce the value slightly and re-check stability. Running right at the edge leaves no headroom for a longer prompt or a larger batch.
- Confirm the split. Load the model and inspect
ollama psbefore trusting the configuration.
Reading ollama ps correctly
The ollama ps command is the authoritative check on what Ollama actually loaded. The FAQ explains the PROCESSOR column: 100% GPU means the model was loaded entirely into the GPU, 100% CPU means it was loaded entirely in system memory, and a split such as 48%/52% CPU/GPU means it was loaded partially onto both.
NAME ID SIZE PROCESSOR CONTEXT UNTIL
gemma4:latest c6eb396dbd59 9.6 GB 100% GPU 131072 2 minutes from nowTwo details in that output matter for tuning. The PROCESSOR column tells you whether offloading worked. The CONTEXT column tells you what context length was actually allocated — which may not match what you intended if a request-level setting, a server default, or a VRAM-based default took precedence. The documentation’s performance guidance is direct: for best performance, use the maximum context length for a model and avoid offloading the model to CPU. In practice those two goals compete, and the resolution depends on your hardware.
Memory estimates and why they mislead
The community tuning guide offers approximate VRAM requirements by model size and quantization. These are useful starting points, not guarantees.
| Model size | Q4 | Q5 | Q8 | FP16 |
|---|---|---|---|---|
| 7B | ~4 GB | ~5 GB | ~7 GB | ~14 GB |
| 13B | ~8 GB | ~9 GB | ~13 GB | ~26 GB |
| 34B | ~20 GB | ~23 GB | ~34 GB | ~68 GB |
| 70B | ~40 GB | ~47 GB | ~70 GB | ~140 GB |
The guide’s own caveat is the important part: actual memory usage also depends on context length and batch size. A 7B Q4 model that fits comfortably at 4k context may not fit at 64k, because the long-context state — the keys and values the transformer retains for every token it has already processed — grows with the number of tokens in the window. Treat the table as a floor for weights, not a ceiling for total memory.
Common symptoms and what they usually mean
- The model loaded but generation is painfully slow. Check
ollama psfirst. A partial split such as48%/52% CPU/GPUmeans a meaningful share of computation is running in system memory. - You raised context and the model fell off the GPU. This is the expected interaction. Larger context consumes memory, leaving less room for layers. Lower the context or lower the offloaded layer count and re-check.
- Out-of-memory errors when increasing
--gpu. This is the documented signal that you have passed the ceiling. Step the value down. - The reported context differs from what you set. Inspect the
CONTEXTcolumn inollama psand confirm which layer set the value — a per-requestnum_ctx, theOLLAMA_CONTEXT_LENGTHserver default, or the app slider.
What to do next
Pick one model you actually use, note its current context and processor split in ollama ps, then change a single variable — context or layer count, not both — and re-measure. Repeat until the processor split stays on the GPU and the context is large enough for your real prompts with a little headroom. Because the right answer depends on your GPU, your quantization, and your workload, a number that works on one machine may be wrong on another; the procedure matters more than any published preset. If you land on a configuration worth sharing, or hit an error you cannot explain, the community is a reasonable place to compare ollama ps output and hardware details.
FAQ
Does a larger context window make the model smarter?
No. Context length controls how much text the model can attend to at once, not its reasoning quality. Ollama's documentation notes that a larger context increases memory requirements, so raising it beyond what your prompt actually needs costs resources without adding capability.
How do I confirm Ollama is using my GPU?
Run ollama ps. The PROCESSOR column shows 100% GPU for a fully GPU-loaded model, 100% CPU for a fully CPU-loaded model, or a split such as 48%/52% CPU/GPU for a partial offload.
Why did generation slow down after I increased num_ctx?
Larger context consumes more memory, which can push model layers off the GPU and into system memory. Check the PROCESSOR column in ollama ps; a partial CPU/GPU split indicates the slowdown source.
What context length should I set for coding or agent workflows?
Ollama's documentation recommends at least 64000 tokens for tasks that require large context, including web search, agents, and coding tools. Whether your hardware can sustain that depends on available VRAM.