Add HF_MODEL to load models directly from huggingface #17801
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem description
Downloading the weights for models is so 2019. We just want to do things like set
HF_MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
and run the demo and have it work first time.What's changed
Add
HF_MODEL
as an alternative to settingLLAMA_DIR
:organisation/model-name
formatLLAMA_CACHE_PATH
if you set it, otherwise inmodel_cache/$HF_MODEL
mistralai/Mistral-7B-Instruct-v0.3
and works out-of-the-box.in0_block_w
selectionWith these changes even more HF models run out-of-the-box!
Checklist