To run Ollama in Docker with GPU access, the container must be given explicit access to the host GPU. On NVIDIA systems, that means installing the NVIDIA Container Toolkit, configuring Docker with nvidia-ctk, and starting the container with --gpus=all. On AMD systems, it means using the rocm image tag and passing the /dev/kfd and /dev/dri devices. Without those steps, Ollama can still run in Docker, but it will use the CPU. The commands below follow the official Ollama Docker documentation and the ollama/ollama Docker Hub image page as retrieved on October 16, 2026.
What GPU access in Docker actually requires
Docker containers are isolated from host hardware by default. A GPU is not automatically visible inside a container just because the host has one. NVIDIA and AMD handle this differently. NVIDIA provides the NVIDIA Container Toolkit, which integrates with Docker so that --gpus=all can expose the GPU. AMD relies on passing specific device files into the container and using an image built for ROCm. The official Ollama Docker image documentation lists separate commands for CPU-only, NVIDIA GPU, AMD GPU, and NVIDIA JetPack systems, and it notes that the AMD path uses the rocm tag.
The default image, ollama/ollama, is the general image. For AMD GPUs, the documented command uses ollama/ollama:rocm. For NVIDIA GPUs, the same default image is used, but the container must be started with GPU access enabled. In all cases, the documented examples publish port 11434 and mount a Docker volume at /root/.ollama so downloaded models persist when the container is replaced.
Prerequisites
- Docker Engine installed and running on the host.
- A supported GPU and working host drivers. NVIDIA systems need NVIDIA drivers; AMD systems need a ROCm-compatible GPU and driver stack.
- For NVIDIA: the NVIDIA Container Toolkit, installed and configured for Docker.
- For AMD: access to the host devices
/dev/kfdand/dev/dri. - For NVIDIA JetPack: the correct JetPack version, because the documentation states that Ollama cannot automatically discover it.
- A terminal with permission to run Docker commands, or
sudowhere required.
CPU-only baseline
If you do not need GPU acceleration, or if you want to confirm that Docker and Ollama work before adding GPU configuration, start with the CPU-only command from the official documentation:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaThis starts a detached container named ollama, publishes the API on port 11434, and stores model data in the ollama volume. A successful start means the container is running and the API is reachable at http://localhost:11434. This command does not provide GPU access.
NVIDIA GPU access
Step 1: Install the NVIDIA Container Toolkit
The official documentation provides package installation commands for Apt-based and Yum or Dnf-based systems. On Apt-based distributions, configure the repository and install the toolkit:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitOn Yum or Dnf-based distributions, the documentation gives the following repository and package commands:
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo yum install -y nvidia-container-toolkitThese commands come from the Ollama Docker documentation. Package names and repository URLs can change, so verify them against the current official documentation if a command fails.
Step 2: Configure Docker to use the NVIDIA runtime
After installing the toolkit, configure Docker and restart the Docker service:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerThis step is what allows Docker to satisfy GPU requests such as --gpus=all. If you skip it, the container may start but fail to see the GPU, or Docker may reject the GPU flag entirely.
Step 3: Start Ollama with GPU access
Start the container with the GPU flag added to the CPU-only command:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaOn an NVIDIA JetPack system, the documentation notes that Ollama cannot automatically discover the correct JetPack version. Pass the environment variable JETSON_JETPACK=5 or JETSON_JETPACK=6 to select version 5 or 6. For example:
docker run -d --gpus=all -e JETSON_JETPACK=6 -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaStep 4: Verify NVIDIA GPU access
Check that the container is running:
docker psThen run a model inside the container. The documented pattern is docker exec -it ollama ollama run ...; replace the placeholder with the model you want to use:
docker exec -it ollama ollama run <model-name>If the model loads and responds, the container is working. To confirm that the GPU is being used rather than the CPU, monitor GPU utilization on the host with your normal NVIDIA tooling while the model is generating output. If utilization stays at zero and inference is slow, revisit the toolkit installation, the nvidia-ctk runtime configure step, the Docker restart, and the --gpus=all flag. The container logs can also reveal startup errors:
docker logs ollamaAMD GPU access
For AMD GPUs, the official command uses the rocm tag and passes the kernel fusion driver and Direct Rendering Infrastructure devices:
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocmThe --device flags expose the host GPU devices to the container. This path does not use --gpus=all. If you use the default ollama/ollama image on an AMD system, you should not expect GPU acceleration; the documented AMD path is the rocm tag. Verification follows the same pattern: confirm the container is running, run a model with docker exec -it ollama ollama run <model-name>, and watch host GPU utilization while the model generates output.
Running a model and checking the API
Once the container is running, Ollama is available inside the container and through the published port. The documented way to run a model interactively is:
docker exec -it ollama ollama run <model-name>The API listens on port 11434. From the host, the base URL is http://localhost:11434. Other containers on the same Docker network can reach the Ollama container by its container name, ollama, on port 11434, provided they share a network. If you only need local access, consider binding the published port to the loopback interface, for example -p 127.0.0.1:11434:11434, instead of publishing it on every host interface. The official examples use -p 11434:11434, which publishes the port more broadly.
The volume ollama:/root/.ollama is important. It keeps downloaded models outside the container lifecycle. If you omit it, models may be lost when the container is removed. If you later pull a newer Ollama image, you can create a new container with the same volume and keep the model data.
Troubleshooting common failures
| Symptom | Likely cause | Fix |
|---|---|---|
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] | The NVIDIA Container Toolkit is missing or Docker has not been configured to use it. | Install the toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker. |
| Container starts, but inference is slow and GPU utilization is near zero. | The GPU was not passed through, or the wrong image is being used for the hardware. | For NVIDIA, add --gpus=all. For AMD, use ollama/ollama:rocm and pass /dev/kfd and /dev/dri. |
Permission denied or missing device errors for /dev/kfd or /dev/dri. | The devices are absent, the driver stack is incomplete, or the Docker process lacks permission to access them. | Confirm the host has a ROCm-compatible GPU and drivers, verify the device files exist, and check host permissions and group membership. |
| JetPack version is not detected. | Ollama cannot automatically discover the correct JetPack version on NVIDIA JetPack systems. | Pass -e JETSON_JETPACK=5 or -e JETSON_JETPACK=6 to select the correct version. |
Port 11434 is already in use. | Another Ollama instance, container, or service is using the port. | Stop the conflicting service or publish a different host port, such as -p 11435:11434. |
| Models disappear after removing the container. | No volume was mounted at /root/.ollama. | Recreate the container with -v ollama:/root/.ollama. |
Limits, tradeoffs, and operational notes
GPU access in Docker is a passthrough configuration, not a performance guarantee. Whether a model fits and how fast it runs depends on the GPU model, available VRAM, the model size and quantization, the context length, and runtime overhead. Model weights are not the only memory cost; the KV cache and runtime overhead also consume memory as context grows. A container that starts successfully can still fail to load a model if the GPU does not have enough memory.
Image tags matter. The default ollama/ollama image is used for CPU-only and NVIDIA GPU examples in the official documentation. AMD uses ollama/ollama:rocm. If you change hardware or move between NVIDIA and AMD, start from the matching documented command rather than assuming the same flags will work. The documentation also mentions Vulkan support bundled into the Ollama image, but the GPU paths covered by the supplied commands are the CPU-only, NVIDIA, AMD ROCm, and JetPack paths; treat other acceleration backends as something to verify against the current official documentation for your platform.
Operationally, keep the Docker volume, bind the API port carefully, and record your exact image tag, GPU model, driver version, and model configuration when you report a problem. Those details are what make a community troubleshooting thread actionable. Because the Ollama image and the NVIDIA Container Toolkit both change over time, a command that worked on one host may need adjustment on another. The official documentation and the Docker Hub image page are the primary references to check first.
A repeatable checklist
- Confirm Docker is installed and running.
- Identify the GPU path: NVIDIA, AMD, or CPU-only.
- For NVIDIA, install the NVIDIA Container Toolkit, run
sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker. - For AMD, confirm ROCm-compatible drivers and device files, then use the
rocmimage tag. - Start the container with the matching documented command and the
ollama:/root/.ollamavolume. - Run a model with
docker exec -it ollama ollama run <model-name>. - Verify GPU utilization on the host while the model generates output.
- If something fails, capture the exact error, image tag, GPU model, driver version, and Docker command before troubleshooting further.
Next actions
Once the container is running, the next useful step is to test the API from your own client, confirm that models persist across container replacement, and monitor GPU memory while increasing context length. If you hit a platform-specific error, share your goal, host OS, GPU model, driver version, Docker image tag, the exact command you ran, and the full error message in the Ai-Forum community so others can compare environments and reproduce the issue.
FAQ
Do I need --gpus=all for AMD GPU access?
No. The documented AMD command uses ollama/ollama:rocm and passes --device /dev/kfd --device /dev/dri instead of --gpus=all.
Where are Ollama models stored in Docker?
Inside the container they are stored under /root/.ollama. The official examples mount the volume ollama:/root/.ollama so models persist when the container is replaced.
Can I run Ollama in Docker without a GPU?
Yes. The CPU-only command is docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. It does not provide GPU acceleration.
Why does Docker say it cannot select a GPU device driver?
That error usually means the NVIDIA Container Toolkit is missing or Docker has not been configured to use it. Install the toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker.
How do I keep models when updating the Ollama container?
Use the same Docker volume, such as ollama:/root/.ollama, when you create the replacement container. The models stored in the volume remain available.