Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
AI and LLM integration

Ollama is slow and the GPU is not being used

Errorollama ps shows 100% CPU or a split such as 48%/52% CPU/GPU

Short answer

Run ollama ps and read the PROCESSOR column: 100% GPU means the model is on the card, anything with CPU means it is not fully there. Then check the usual blocks: a driver older than Ollama needs, a GPU lost after suspend, a Docker container started without GPU access, or an AMD card that needs an override.

  • Applies to Ollama 0.x, NVIDIA driver 550 and later, Docker 27, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Load a model, then ask Ollama where it put it. The PROCESSOR column is the answer.

    ollama run llama3.2 "hello"
    ollama ps
  2. 2Match what you see, then jump to its fix

    1. 1ollama ps shows CPU only on a supported cardThe NVIDIA driver is too old for OllamaFix
    2. 2It was fast before the laptop or server slept, and is CPU-only sinceThe GPU disappeared after suspend and resume (Linux, NVIDIA)Fix
    3. 3Ollama works in Docker but only on CPUThe Docker container was started without GPU accessFix
    4. 4Ollama runs on CPU with a Radeon card (Linux)An AMD card Ollama does not recognise, or lacks permission to useFix

What does the error look like?

  • ollama ps shows 100% CPU or a split such as 48%/52% CPU/GPU
  • Replies arrive a few tokens per second on a machine with a GPU

Cause 1 of 4

The NVIDIA driver is too old for Ollama

1Check

Ollama needs compute capability 5.0 or higher and driver 550 or newer. Cards with compute capability 5.0 to 6.2 need driver 570 or newer.

nvidia-smi --query-gpu=name,driver_version --format=csv

2Fix

Update the NVIDIA driver to a version that meets the requirement, then restart Ollama.

sudo systemctl restart ollama

3Verify

Load the model again and look at the PROCESSOR column.

ollama run llama3.2 "hello"
ollama ps

Confirmed by docs.ollama.com

Did this fix it?

Cause 2 of 4

The GPU disappeared after suspend and resume (Linux, NVIDIA)

1Check

Ollama's GPU guide lists this case: the UVM driver must be reloaded after resume.

nvidia-smi -L

2Fix

Reload the nvidia_uvm kernel module, then restart Ollama.

sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
sudo systemctl restart ollama

3Verify

Run a model and read ollama ps again.

ollama ps

Confirmed by docs.ollama.com

Did this fix it?

Cause 3 of 4

The Docker container was started without GPU access

1Check

The CPU-only run command has no --gpus flag. NVIDIA GPUs also need the NVIDIA Container Toolkit configured for Docker.

docker inspect ollama --format '{{.HostConfig.DeviceRequests}}'

2Fix

Configure Docker for the NVIDIA runtime, restart Docker, and recreate the container with --gpus=all. For AMD use the rocm image with the /dev/kfd and /dev/dri devices.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker rm -f ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

3Verify

Run a model inside the container and check where it loaded.

docker exec -it ollama ollama run llama3.2 "hello"
docker exec -it ollama ollama ps

Confirmed by docs.ollama.com

Did this fix it?

Cause 4 of 4

An AMD card Ollama does not recognise, or lacks permission to use

1Check

Supported AMD cards need the ROCm v7 driver. For a card outside the supported list, Ollama documents an override, and on some distributions the ollama user must be in the render group.

rocminfo | grep -i gfx

2Fix

Add the ollama user to the render group. If the card is unsupported, try HSA_OVERRIDE_GFX_VERSION with a close supported version, for example 10.3.0, set on the Ollama service.

sudo usermod -aG render ollama
sudo systemctl edit ollama
# in the editor add:
# [Service]
# Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
sudo systemctl restart ollama

3Verify

Run a model and read the PROCESSOR column.

ollama ps

Confirmed by docs.ollama.com

Did this fix it?

How do I stop it coming back?

  • Look at ollama ps after every model change, not only when it feels slow.
  • Keep the GPU driver at or above the version Ollama lists, and recheck after OS upgrades.
  • Keep the Docker run command in a compose file or script so the GPU flag is never forgotten.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

AI

LlamaIndex cannot import Ollama from llama_index.llms

ImportError: cannot import name 'Ollama' from 'llama_index.llms' (unknown location)

Since LlamaIndex 0.10, each integration is its own package and the old top-level llama_index.llms path is gone.

3 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.