Skip to content

Conversation

@nogeonwoo
Copy link

This PR adds support for Apple Silicon (MPS), enabling GPU acceleration for users on modern Macs.
When trying to run 4-bit quantized GGUF models like calcuis/higgs-gguf on an Apple Silicon Mac, the only option was to use the CPU, which was quite slow. This change allows the model to utilize the MPS backend for a significant performance improvement, making the project much more usable for Mac users.

  • Modified the device detection logic to include a check for torch.backends.mps.is_available().
  • The device selection priority is now: CUDA -> MPS -> CPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant