Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HF_MODEL to load models directly from huggingface #17801

Merged
merged 5 commits into from
Feb 12, 2025
Merged

Conversation

yieldthought
Copy link
Contributor

@yieldthought yieldthought commented Feb 10, 2025

Problem description

Downloading the weights for models is so 2019. We just want to do things like set HF_MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and run the demo and have it work first time.

What's changed

Add HF_MODEL as an alternative to setting LLAMA_DIR:

  • Loads the model from HuggingFace directly using their organisation/model-name format
  • Creates tenstorrent cache tensor files in the existing LLAMA_CACHE_PATH if you set it, otherwise in model_cache/$HF_MODEL
  • Tested with mistralai/Mistral-7B-Instruct-v0.3 and works out-of-the-box.
  • Work around as_tensor issue to enable models with bias to run again on N150
  • Generalise 2d matmul in0_block_w selection
    With these changes even more HF models run out-of-the-box!

Checklist

Copy link
Contributor

@mtairum mtairum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@mtairum
Copy link
Contributor

mtairum commented Feb 11, 2025

Double check the accuracy numbers, the pipeline was failing due to that.

@mtairum mtairum force-pushed the yieldthought/hf-url branch from 6fb0f58 to 9adc481 Compare February 11, 2025 17:47
@mtairum
Copy link
Contributor

mtairum commented Feb 11, 2025

Rebased and updated Perf.md.

Re-running:

@mtairum mtairum merged commit d2f0b15 into main Feb 12, 2025
217 of 218 checks passed
@mtairum mtairum deleted the yieldthought/hf-url branch February 12, 2025 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants