As of March 2024, ollama in nixpkgs manages to handle CUDA mess on its own.
A sensible nixos module is available too! There is no need to use this
flake anymore.

Original README content:
As of 12 Nov 2023, ollama in nixpkgs fails to utilize CUDA-enabled devices.

This repo contains a nix flake that defines:
- ollama package with CUDA support
- NixOS module that may be helpful if you plan on using ollama as a system-wide
  LLM runner

To simply run the CUDA-enabled ollama binary, make sure you have nvidia-smi
available in the PATH and run the following command:
nix run github:havaker/ollama-nix