mtmd : add qwen2vl and qwen2.5vl

[ngxson]

Tested with:

llama-mtmd-cli -hf bartowski/Qwen2-VL-2B-Instruct-GGUF
llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF

NOTE: the Qwen2-VL-2B seems to give poor result on llama-qwen2vl-cli, could be due to incorrect n_past tracking, see test result below ; it's now giving the correct answer in llama-mtmd-cli

# WRONG result with llama-qwen2vl-cli
llama-qwen2vl-cli -hf bartowski/Qwen2-VL-2B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# I'm sorry, but I cannot see any image or any people in this context. Can you provide more details or clarify what you are asking?


# CORRECT result with llama-mtmd-cli
llama-mtmd-cli -hf bartowski/Qwen2-VL-2B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# There is one person in the image. The composition of the image is a close-up of the person wearing a hat.

llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# There is one person in the image. The composition of the image is a close-up of the person's face, focusing on their eyes and the hat they are wearing.

---

Test results:

OK:   llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
OK:   llama-mtmd-cli guinmoon/MobileVLM-3B-GGUF:Q4_K_M
OK:   llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M
OK:   llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
OK:   llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K
OK:   llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
OK:   llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
OK:   llama-mtmd-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M

[ngxson]

@HimariO llama-qwen2vl-cli seems to give incorrect response with Qwen2VL:

llama-qwen2vl-cli -hf bartowski/Qwen2-VL-2B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# I'm sorry, but I cannot see any image or any people in this context. Can you provide more details or clarify what you are asking?

I suspect this is due to the st_pos_id being tracked incorrectly for image. As I understand, one image is correspond to 1 temporal position instead of max(pw, ph):

llama.cpp/examples/llava/qwen2vl-cli.cpp

Line 53 in 4e87962

*st_pos_id += std::max(pw, ph);

In the implementation introduced in this PR, I only advance n_past by 1 for each image, which now giving correct result.

So just want to double-check with you if this way is correct. Thanks!

See comment below

[ngxson]

@HimariO Please note that llama-qwen2vl-cli is no longer used, the target is deprecated in cmake. I still keep this file because it contains your code for testing M-RoPE

[ngxson]

I suspect this is due to the st_pos_id being tracked incorrectly for image. As I understand, one image is correspond to 1 temporal position instead of max(pw, ph):

Small update, if I change the mentioned line in qwen2vl-cli.cpp to:

*st_pos_id += 1;

Then, Qwen2-VL-2B-Instruct works correctly.

---

Edit: it could be related to #13159

[ggerganov]

Rename img_h and img_w to nx and ny and fix calls:

clip_embd_nbytes_by_img(ctx, ny, nx); <-- wrong
clip_embd_nbytes_by_img(ctx, nx, ny);

[ngxson]

Fixed in 496f1ce

[ggerganov]

This is the other call.

[ngxson]

Looking at the source code of transformers, I think it's correct to have 1 position per image instead of std::max(pw, ph), otherwise it make no sense to process video inputs. So I'm keeping my version as-is.

https://github.com/huggingface/transformers/blob/4602059aaee7b075ca51c21941ab2d27980bef7c/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1460-L1471

This is also confirmed from the illustration from their paper:

If someone knows more about this, feel free to correct that.

[ngxson]

Running test again, all is good. So I'm merging this PR once the CI is green

OK:   llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
OK:   llama-mtmd-cli guinmoon/MobileVLM-3B-GGUF:Q4_K_M
OK:   llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M
OK:   llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
OK:   llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K
OK:   llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
OK:   llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
OK:   llama-mtmd-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M

llama-mtmd-cli -hf bartowski/Qwen2-VL-2B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# There is one person in the image. The composition of the image is a close-up of the person wearing a hat.

llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF --image ../models/lenna.png -p "How many people do you see in the image? What is the composition of this image?"
# There is one person in the image. The composition of the image is a close-up of the woman's face, focusing on her eyes and the hat she is wearing.