feat: add initial qwen2.5-vl model and test

[drbh]

This PR adds support for qwen2.5-vl models and currently loads the weights and supports reasonable responses. Opening early for exposure and any feedback.

These changes are dependent on #2943 and must be rebased/merged after it is merged

items

• allow qwen2_5_vl models to load in server/router
• add updated vision model
  • adjust weight names/loading
  • adjust config naming
  • RMSNorm in vision blocks
  • update MLP for up/gate/down
  • add fullatt_block_indexes
• validation
  • add tests
  • add snapshots
  • tests more outputs locally for edge cases
  • inspect logits for correctness
  • ensure mrope is producing correct ids

small reproducible example:

text-generation-launcher --model-id Qwen/Qwen2.5-VL-3B-Instruct

script

import requests
import json

url = "http://127.0.0.1:3000/generate"
headers = {"Content-Type": "application/json"}
image_urls = [
    "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/rabbit.png",
]

for image in image_urls:
    query = "Describe the image"
    payload = {
        "inputs": f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n![]({image}){query}<|im_end|>\n<|im_start|>assistant\n",
        "parameters": {"max_new_tokens": 50},
    }
    response = requests.post(url, headers=headers, json=payload)
    print(json.dumps(response.json(), indent=4))

output

{
    "generated_text": "The image showcases the iconic Statue of Liberty in New York City, with the New York City skyline in the background. The statue is a large, green-colored sculpture on a stone pedestal, with the American flag on a flagpole in the foreground. The"
}
{
    "generated_text": "The image depicts a character in a space suit, set in a rocky, desert-like environment with a warm, orange hue. The character is a large, brown, and white rabbit with long, pointed ears and a small, beak-like nose."
}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Narsil]

LGTM!

[Narsil]

Do we still need that with 4.49 release ?

[drbh]

this should be able to be removed with 4.49

[Narsil]

Is this not defined in transformers ?

[drbh]

this was added in 4.49 so can be be removed with the update too

[Narsil]

Suggested change

	if (
	self.model.config.model_type == "qwen2_vl"
	or self.model.config.model_type == "qwen2_5_vl"
	):
	if self.model.config.model_type in {"qwen2_vl", "qwen2_5_vl"}:

Fixing the chatbot for you :)

[drbh]

wish I could blame it on a bot 😅

thanks!

[Narsil]

NIT: Simpler condition.