Eval bug: Control vectors for Qwen3VL fail due to n_embd size

[65a]

Name and Version

version: 6906 (0de0a01)
built with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu

Operating systems

Linux

GGML backends

HIP

Hardware

probably not relevant

Models

Qwen3VL

Problem description & steps to reproduce

Not sure if this is a bug or I am holding it wrong.

Apparently Qwen3VL uses n_embd strangely in llama.cpp. Computing a control vector as I do for other models produces a result with a size of 5120 per layer, but I cannot load it.

Although the hidden_size is 5120 in transformers, llama.cpp asserts unless the vector is 20480. Padding the remaining values, or commenting out the assert results in a different error, GGML_ASSERT(ggml_can_repeat(b, a)).

Reading the merge CL for Qwen3VL, it sounds like n_embd is extended to handle some recurrent stuff. Is a different approach to calculate control vectors necessary here? How should this work for Qwen3VL?

First Bad Commit

n/a

Relevant log output

apply: control vector n_embd does not match model
srv    load_model: failed to load model, '.../path/to/model.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

[ggerganov]

Yeah, I think the scaling of hparams.n_embd might have been a bit overlooked. Think we should keep the meaning of hparams.n_embd as it is and find an alternative way to represent the deepstack concat.

llama.cpp/src/llama-model.cpp

Lines 1042 to 1044 in 5d8bb90

	// since vision model stacks deepstack features along feature dim
	// we also create a fake "n_embd" for text model to be the main embd + deepstack embds
	hparams.n_embd *= hparams.n_deepstack_layers + 1;

cc @CISC

[CISC]

I flagged it here #16780 (comment)

[CISC]

It should be as simple as maintaining an n_embd_full where needed.

[CISC]

It should be as simple as maintaining an n_embd_full where needed.

Sigh, or not...

[65a]

I think I have this locally, but didn't test everything, just text. It looks like n_embd_full was created, but there are other places where it's assumed that n_embd should be divided by the deepstack layer count etc. I can try to do a pull for it.

[CISC]

I have something working, but it's possibly a little half baked, I'll make a PR to discuss it.

[65a]

My pull wasn't as comprehensive I think, since it had to recover the larger size wherever it was used rather than storing in hparams. Dropped mine.