ValueError when Loading Qwen2-VL Model with Liger Kernel

[rahatarinasir]

🐛 Describe the bug

I'm encountering a ValueError when trying to load the Qwen2-VL model using the AutoLigerKernelForCausalLM class from the Liger Kernel. The error message indicates an unrecognized configuration class for this model.

Reproduce

Clone the repository:
bash

git clone https://github.com/linkedin/Liger-Kernel.git
cd Liger-Kernel
pip install -e .

Install necessary packages:
python

from huggingface_hub import login
hf_token = input("Enter your Hugging Face token: ")
login(token=hf_token)
pip install torch
pip install triton # >= 2.3.0
pip install liger-kernel
pip install liger-kernel-nightly
pip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830

Attempt to load the model:
python

from transformers import Qwen2VLConfig, Qwen2VLForConditionalGeneration
from liger_kernel.transformers import AutoLigerKernelForCausalLM

config = Qwen2VLConfig.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")
model = AutoLigerKernelForCausalLM.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", config=config)

Error Message
angelscript

ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_vl.configuration_qwen2_vl.Qwen2VLConfig'> for this kind of AutoModel: AutoLigerKernelForCausalLM.
Model type should be one of [list of model types].

It seems that the AutoLigerKernelForCausalLM class does not recognize the Qwen2VLConfig. Is there a workaround or an update planned to support this model? Any guidance would be greatly appreciated!

Versions

Environment Report:

Operating System: Linux-6.1.85+-x86_64-with-glibc2.35
Python version: 3.10.12
PyTorch version: 2.4.0+cu121
CUDA version: 12.1
Triton version: 3.0.0
Transformers version: 4.45.0.dev0

[tyler-romero]

Qwen2-VL isnt based off of AutoModelForCausalLM, so it cant be loaded with AutoLigerKernelForCausalLM. Instead its a "ForConditionalGeneration" model, which doesn't have an AutoModel parent class.

As a workaround, I'd recommend using the monkeypatch function apply_liger_kernel_to_qwen2_vl after from transformers import Qwen2VLForConditionalGeneration