Reduction in the number of parameters

[sri-fiddler]

The number of parameters for a Llama 3.1 8B model is 8.03B according to Huggingface. The same Unsloth model, when quantised to 4bits is 4.56B. How and why is there a reduction in the number of parameters? As far as I am aware, quantisation reduces the size of each parameter, but it doesn't do away with it.

Meta Llama 3.1 8B : https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Unsloth Llama 3.1 8B : https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit

[danielhanchen]

Oh not a bug - it's because the pre-quantized uploads saves it directly in 4bit, and packs them into 1 8bit number, hence the reduction

[sri-fiddler]

This is quite brilliant. Good optimisation.