Skip to content

feat: add GPU support to metaflow-dev minikube setup #2609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

feat: add GPU support to metaflow-dev minikube setup #2609

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc75b62
feat: add GPU support to metaflow-dev minikube setup
cnaples79 Sep 17, 2025
20ec83b
Address Docker compatibility feedback for GPU support
cnaples79 Sep 17, 2025
8ca6ae4
minikube GPU: remove unsupported --devices flag; always use --gpus al…
cnaples79 Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions devtools/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ help:
@echo " down - Stop and clean up the environment"
@echo " all-up - Start the development environment with all services"
@echo " help - Show this help message"
@echo ""
@echo "Environment variables:"
@echo " MINIKUBE_ENABLE_GPU=auto|true|false - Control GPU support (default: auto)"

HELM_VERSION := v3.14.0
MINIKUBE_VERSION := v1.32.0
Expand All @@ -31,6 +34,7 @@ MAKE_CMD := $(MAKE) -f "$(MKFILE_PATH)"
MINIKUBE_CPUS ?= 4
MINIKUBE_MEMORY ?= 6144
MINIKUBE_DISK_SIZE ?= 20g
MINIKUBE_ENABLE_GPU ?= auto

ifeq ($(shell uname), Darwin)
minikube_os = darwin
Expand All @@ -48,6 +52,24 @@ else
tilt_arch = arm64
endif

# GPU detection for minikube
ifeq ($(MINIKUBE_ENABLE_GPU), auto)
# Auto-detect GPU availability
ifeq ($(shell command -v nvidia-smi >/dev/null 2>&1 && echo "nvidia"), nvidia)
gpu_flags = --gpus all
Copy link

@feltech feltech Sep 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on how Docker is configured, --gpus all might not work, and instead you need to use --devices nvidia.com/gpu=all or similar.

(I had this problem with Docker on NixOS)

Copy link
Author

@cnaples79 cnaples79 Sep 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback, I'll update.

Copy link

@feltech feltech Sep 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, I may have misled you here, the problem I referred to was using the docker command-line, rather than the minikube command-line. They both accept a --gpus argument, but I think minikube is a bit more clever about it. Indeed, --devices doesn't seem to be valid for minikube:

$ minikube start --devices nvidia.com/gpu=all
Error: unknown flag: --devices

My bad, got my wires totally crossed.

Copy link
Author

@cnaples79 cnaples79 Sep 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh I see. No worries, I'll update the PR with the correct fix.

else ifeq ($(shell command -v rocm-smi >/dev/null 2>&1 && echo "amd"), amd)
gpu_flags = --gpus all
else
gpu_flags =
endif
else ifeq ($(MINIKUBE_ENABLE_GPU), true)
# Force enable GPU support
gpu_flags = --gpus all
else
# Explicitly disabled
gpu_flags =
endif

# TODO: Move scripts to a folder

install-helm:
Expand Down Expand Up @@ -136,11 +158,13 @@ setup-minikube:
@echo "🔧 Setting up Minikube $(MINIKUBE_VERSION) cluster..."
@if ! $(MINIKUBE) status >/dev/null 2>&1; then \
echo "🚀 Starting new Minikube $(MINIKUBE_VERSION) cluster..."; \
if [ -n "$(gpu_flags)" ]; then echo "🎮 GPU support detected - enabling GPU passthrough"; else echo "💻 No GPU detected - starting without GPU support"; fi; \
$(MINIKUBE) start \
--cpus $(MINIKUBE_CPUS) \
--memory $(MINIKUBE_MEMORY) \
--disk-size $(MINIKUBE_DISK_SIZE) \
--driver docker \
$(gpu_flags) \
|| { echo "❌ Failed to start Minikube (check if Docker is running)"; exit 1; }; \
echo "🔌 Enabling metrics-server and dashboard (quietly)..."; \
$(MINIKUBE) addons enable metrics-server >/dev/null 2>&1; \
Expand Down