Draft: Merge LoRA Adapters with AWQ BaseModels

[Whadup]

This PR extends the AwqLoraLinear class to allow merging in of LoRA Adapters.
Instead of re-quantizing the whole model, we use the original quantization scales and zeros.

[BenjaminBossan]

Thanks for adding merging capabilities to AWQ. I only skimmed the PR so far, but could you please:

• Also implement the unmerge method? It should be very similar to the merge method, but remove the delta weight
• There should be a unit test to ensure that merging works, e.g. similar to this test (without DoRA).
• Let's run make style on your changes.

[Whadup]

@BenjaminBossan Thanks for looking into it already! Your three points are on my agenda, I will give you a ping when I commit the changes.

[BenjaminBossan]

Great, thanks a lot.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

@Whadup Do you still plan on working on this?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

It's not quite clear to me, but it appears like AutoAWQ will be integrated into llm-compressor:

AutoAWQ Integration: Perform low-bit weight-only quantization efficiently using AutoAWQ, now part of LLM Compressor. Note: This integration should be considered experimental for now.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.