AIForum.

How to Run a Local LLM with GPU Acceleration in Docker

On this page

THE SHORT ANSWER

GPU acceleration in Docker is mostly a plumbing problem: install the NVIDIA Container Toolkit, then pass the GPU into the container explicitly. This guide covers the two official images that handle most local LLM workloads, the shared-memory detail that breaks first attempts, and an ordered diagnostic path when the GPU is not visible.
Quick facts

Running a large language model on your own hardware with GPU acceleration inside Docker is mostly a plumbing problem, not a modelling problem. Containers are isolated from the host by default, so a container cannot see your NVIDIA GPU unless the NVIDIA Container Toolkit is installed and the GPU is explicitly requested with flags such as --gpus all. Once that plumbing works, two official images cover most practical needs: vLLM’s vllm/vllm-openai image, which serves an OpenAI-compatible API, and Ollama’s container, which is simpler to operate for single-user local work.

This guide is for technically curious beginners and intermediate practitioners who already have a machine with an NVIDIA GPU and want a reproducible, container-based setup. It covers prerequisites, two working deployment paths, the shared-memory detail that trips up most first attempts, and an ordered diagnostic path for when the GPU is not visible.

What GPU acceleration in Docker actually requires

A container ships with its own userland and, by default, no device access. GPU support therefore needs three layers working together: a host NVIDIA driver, the NVIDIA Container Toolkit (which includes the NVIDIA Container Runtime and a plugin for Docker), and a run command that requests the GPU. The toolkit’s installation guidance stresses installing the GPU drivers first, then enabling the NVIDIA CUDA repository before installing the toolkit packages [2].

On Debian and Ubuntu systems, the repository setup documented for the toolkit looks like this:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee 
  /etc/apt/sources.list.d/nvidia-docker.list

After installing the toolkit and restarting Docker, a container started with --gpus all should be able to run nvidia-smi and list the host GPU. Treat that as the first checkpoint: if nvidia-smi fails inside a container, no amount of model configuration will help, because the fault sits in the driver or toolkit layer rather than in vLLM or Ollama.

Choosing between the vLLM and Ollama images

Both images run local models on an NVIDIA GPU, but they optimise for different jobs.

  • vLLM (vllm/vllm-openai) is the official image for running an OpenAI-compatible server, published on Docker Hub. It is the better fit when you want an HTTP endpoint that other tools can point at, and it is built with tensor-parallel inference in mind [1].
  • Ollama packages model download, management, and a local API. Its Docker image has existed since 2023, and the documented workflow covers both a one-line docker run command and a Docker Compose file, with NVIDIA GPU support enabled through the container toolkit [2].

A practical rule: if you are serving one user on one GPU and want the least ceremony, start with Ollama. If you need an OpenAI-compatible endpoint, batching, or multi-GPU tensor parallelism, start with vLLM.

Path A: vLLM’s official OpenAI-compatible image

The documented invocation passes the GPU through, mounts the Hugging Face cache so downloaded weights survive container restarts, forwards port 8000, and shares host memory [1].

docker run --runtime nvidia --gpus all 
  -v ~/.cache/huggingface:/root/.cache/huggingface 
  --env "HF_TOKEN=$HF_TOKEN" 
  -p 8000:8000 
  --ipc=host 
  vllm/vllm-openai:latest 
  --model Qwen/Qwen3-0.6B

Three details are worth noting before you adapt this command.

  1. Engine arguments go after the image tag. Anything you would normally pass to the vLLM server, such as --model, is appended after vllm/vllm-openai:latest, not before it [1].
  2. The Hugging Face token is optional but often necessary. Gated repositories will fail to download without HF_TOKEN, and the volume mount keeps the cache between runs so weights are not re-downloaded each time [1].
  3. Shared memory matters. vLLM uses PyTorch, which relies on shared memory to pass data between processes, particularly for tensor-parallel inference. The documentation says you can either use --ipc=host or set --shm-size to give the container access to the host’s shared memory [1].

Once the server is up, the observable check is an HTTP request against the OpenAI-compatible endpoint on port 8000, for example curl http://localhost:8000/v1/models. A JSON list containing your model name means the container, the GPU, and the server are all working together.

Podman users are not excluded: the same documentation shows the equivalent invocation using --device nvidia.com/gpu=all in place of the Docker runtime flags [1].

When the model needs dependencies the image does not ship

vLLM’s pre-built image deliberately omits optional dependencies to avoid licensing issues. If you have accepted the relevant licence terms and need them, the documented approach is a thin custom Dockerfile on top of the base image, keeping the vLLM version aligned between the base tag and the extra install [1]:

FROM vllm/vllm-openai:v0.11.0
RUN uv pip install --system vllm==0.11.0

The same pattern handles brand-new models that need the development version of Hugging Face Transformers [1]:

FROM vllm/vllm-openai:latest
RUN uv pip install --system git+https://github.com/huggingface/transformers.git

This is also the honest answer to a common licensing question: open weights, open source, and commercial-use permission are three different things, and the image’s dependency choices reflect licensing constraints rather than technical limits [1].

Path B: Ollama in Docker with NVIDIA GPU

The prerequisite is identical: the NVIDIA Container Toolkit, installed after the GPU driver, so that Docker containers can reach the GPU [2]. With that in place, the toolkit documentation describes two equivalent approaches, a single docker run command and a Docker Compose file [2].

The one-liner form follows the usual pattern of requesting all GPUs, persisting the model store in a named volume, and publishing Ollama’s default port:

docker run -d --gpus=all 
  -v ollama:/root/.ollama 
  -p 11434:11434 
  --name ollama 
  ollama/ollama

After the container starts, pull a model inside it and confirm the GPU is being used. The observable checks are a successful model pull and a response from the local API on port 11434. If the model runs but generation is unexpectedly slow, the container is probably falling back to CPU, which points back to the toolkit or the --gpus flag rather than to Ollama itself.

Compose is the better choice when you want the configuration in version control, since the GPU reservation, volume, and port mapping live in a file instead of a shell history [2].

Diagnosing the usual failures in order

Work from the outside in. Each step below eliminates an entire layer, so you do not end up tuning model parameters to fix a driver problem.

  1. Is the GPU visible on the host? Run nvidia-smi outside Docker. If it fails, fix the driver first.
  2. Is the toolkit installed and is Docker aware of it? A container started with --gpus all that cannot run nvidia-smi indicates a missing or misconfigured NVIDIA Container Toolkit, not a model problem [2].
  3. Did you pass the GPU flag? --runtime nvidia --gpus all in the vLLM example, --gpus=all in the Ollama example [1][2].
  4. Are you hitting shared-memory limits? Symptoms tend to appear under load or with tensor parallelism. Add --ipc=host or raise --shm-size as the vLLM documentation recommends [1].
  5. Is the model download failing? Gated repositories need HF_TOKEN, and the cache mount must point somewhere writable [1].
  6. Is the container out of GPU memory? Reduce context length, choose a smaller or more quantised model, or lower concurrency. Weight size alone is not the whole budget.
  7. Is the driver older than the CUDA build? vLLM’s image ships CUDA compatibility libraries specifically to help with older CUDA drivers [1], but that is a mitigation, not a guarantee, so check the notes for your image version.

Sizing expectations before you commit

The most common planning mistake is treating the model file size as the memory requirement. In practice the GPU must hold the weights plus the KV cache for the context you actually use, plus runtime overhead. Context length is a direct multiplier on KV cache, so a configuration that fits at a short context can fail at a long one on identical hardware. Quantisation reduces the weight portion, not the cache portion. Because these figures depend on the model architecture, the quantisation format, and your context settings, the reliable approach is to measure on your own hardware rather than to trust a general table.

A sensible first run

Validate the plumbing with a small model before downloading anything large. Start the vLLM image with a small model, confirm /v1/models responds, then swap in the model you actually want. If you prefer the simpler operational model, do the same with Ollama: confirm the container responds on port 11434 and that GPU utilisation appears during generation. Only after both checkpoints pass is it worth tuning batching, context length, or multi-GPU tensor parallelism.

If you hit a wall, the most useful thing you can share with the community is your environment: host OS and driver version, whether nvidia-smi works inside a container, the exact image tag, the full run command, and the model identifier. That combination usually identifies the failing layer in one reply.

FAQ

Do I still need the NVIDIA Container Toolkit if the GPU driver is already installed?

Yes. The driver lets the host use the GPU; the toolkit is what exposes that GPU to Docker containers. The documented installation order is driver first, then the toolkit [2].

Why does the vLLM example use --ipc=host?

vLLM is built on PyTorch, which uses shared memory to pass data between processes, particularly for tensor-parallel inference. The documentation allows either --ipc=host or an explicit --shm-size value [1].

Can I run these images with Podman instead of Docker?

Yes. The vLLM documentation shows the Podman equivalent using --device nvidia.com/gpu=all in place of the Docker runtime flags [1].

Why are some optional dependencies missing from the vLLM image?

They are excluded to avoid licensing issues. If you have accepted the relevant licence terms, the documented fix is a custom Dockerfile on top of the base image that installs them, keeping the vLLM version matched to the base tag [1].

The model runs but generation is very slow. What should I check first?

Confirm the container actually has GPU access by running nvidia-smi inside it, then verify the GPU flag in your run command. A container without GPU access falls back to CPU, which looks like a performance problem rather than a configuration one [1][2].

References

  1. https://docs.vllm.ai/en/stable/deployment/docker/
  2. https://itsfoss.com/ollama-docker/

Related terms

NVIDIA Container Toolkit
The package that includes the NVIDIA Container Runtime and a Docker plugin, allowing containers to access host GPUs.
OpenAI-compatible server
An HTTP service exposing endpoints such as /v1/models and /v1/chat/completions so existing OpenAI client libraries can point at a local model.
KV cache
GPU memory used to store attention keys and values for the tokens in the current context; it grows with context length and concurrency.
Tensor parallelism
Splitting a model's tensor operations across multiple GPUs, which is one reason vLLM relies on shared memory between processes.
CUDA compatibility libraries
Libraries bundled into vLLM's Docker image to help it run on systems with older CUDA drivers.

Leave a Reply

Your email address will not be published. Required fields are marked *