Added Support for Custom Quantization #35915
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Hi @keetrap, Thanks for PR ! It looks great and it's a very handy feature for the community ! I just left some very small nits |
custom_quant_example.py
Outdated
| @register_quantization_config("custom") | ||
| class CustomConfig(QuantizationConfigMixin): | ||
| def __init__(self): | ||
| self.quant_method = "custom" | ||
| self.bits = 8 | ||
|
|
||
| def to_dict(self) -> Dict[str, Any]: | ||
| output = { | ||
| "num_bits": self.bits, | ||
| } | ||
| return output | ||
|
|
There was a problem hiding this comment.
I am not sure if the example should be here, maybe we can add some doc about it, wdyt @SunMarc
There was a problem hiding this comment.
Let's put it in the transformers/examples/quantization folder. Also can you try to do a complete example e.g implementing a 8-bit quantization ?
There was a problem hiding this comment.
Can you also add a doc about this new feature in the quantization ?
Maybe add a quick description in the overview docs and potentially update the following doc or create a new section called Custom Quantization.
There was a problem hiding this comment.
Hey, I tried to add a complete example for 8-bit quantization, and it seems to be working fine as far as I know. However, since I’m still learning, it might be better if someone with more experience could add the example.
It would be easier to add the documentation if there's a complete working example available, as we could reference that in the docs. However, if you'd prefer me to continue with my example and create the documentation based on that, I will do it. Just let me know how you'd like to proceed.
There was a problem hiding this comment.
Try to do a complete example for 8-bit quantization ! We will help you review it if necessary =)
src/transformers/modeling_utils.py
Outdated
| try: | ||
| user_agent["quant"] = hf_quantizer.quantization_config.quant_method.value | ||
| except Exception: | ||
| user_agent["quant"] = hf_quantizer.quantization_config.quant_method | ||
| # Force-set to `True` for more mem efficiency |
There was a problem hiding this comment.
Maybe it's better to use an if/else statement to avoid capturing unintended exceptions, wdyt ?
src/transformers/modeling_utils.py
Outdated
| try: | ||
| user_agent["quant"] = hf_quantizer.quantization_config.quant_method.value | ||
| except Exception: | ||
| user_agent["quant"] = hf_quantizer.quantization_config.quant_method |
There was a problem hiding this comment.
instead of a try except, you can just check if value attribute exist or not
There was a problem hiding this comment.
I will update this.
custom_quant_example.py
Outdated
| @register_quantization_config("custom") | ||
| class CustomConfig(QuantizationConfigMixin): | ||
| def __init__(self): | ||
| self.quant_method = "custom" | ||
| self.bits = 8 | ||
|
|
||
| def to_dict(self) -> Dict[str, Any]: | ||
| output = { | ||
| "num_bits": self.bits, | ||
| } | ||
| return output | ||
|
|
There was a problem hiding this comment.
Let's put it in the transformers/examples/quantization folder. Also can you try to do a complete example e.g implementing a 8-bit quantization ?
custom_quant_example.py
Outdated
| @register_quantization_config("custom") | ||
| class CustomConfig(QuantizationConfigMixin): | ||
| def __init__(self): | ||
| self.quant_method = "custom" | ||
| self.bits = 8 | ||
|
|
||
| def to_dict(self) -> Dict[str, Any]: | ||
| output = { | ||
| "num_bits": self.bits, | ||
| } | ||
| return output | ||
|
|
There was a problem hiding this comment.
Can you also add a doc about this new feature in the quantization ?
Maybe add a quick description in the overview docs and potentially update the following doc or create a new section called Custom Quantization.
|
@SunMarc @MekkCyber @keetrap Sorry I'm just seeing this now - I've been on Chinese New Year vacation. I ran into a problem with the QuantizationMethod enum not being extensible, so I couldn't come up with a good solution. I think this PR is impressive and aligns well with my expectations. LGTM! |
|
@SunMarc @MekkCyber |
|
Sorry for the delay @keetrap, the CI is green will merge now |
This PR adds a new feature to support custom quantization in the Transformers library.
Closes #35814
@SunMarc @MekkCyber