[hmellor]

Overview

Serve Llama with the Transformers modeling backend. vllm/model_executor/models/llama.py moves to vllm/model_executor/layers/llama.py, kept only for the models that still build on the native Llama classes. Draft until the gaps below are closed. Tracked in #67003.

Claims

• LlamaForCausalLM, LLaMAForCausalLM, LlamaModel and MistralModel resolve to the Transformers modeling backend.
• InternLM3ForCausalLM and IQuestCoderForCausalLM stay native here, importing from vllm/model_executor/layers/llama.py. [Transformers backend] Migrate InternLM3 and IQuestCoder #66416 moves them. [Model] Remove TeleChat2 and TeleChat3 #67035 removes TeleChat2ForCausalLM and TeleChat3ForCausalLM.
• LlamaBidirectional* stay native here, [Transformers backend] Serve bidirectional Llama with the Transformers modeling backend #67018 moves them.
• Every module that imported from llama.py (Mistral, Llama 4, Eagle/Eagle3, GLM4, MiniCPM-V, ...) imports from vllm/model_executor/layers/llama.py instead.
• VLMs built with init_vllm_registered_model call self.language_model(...) instead of self.language_model.model(...), so their text model can be a Transformers modeling backend model (Llava-1.5 fails at startup without this).
• Phi3-V's text model is the backend's Phi3ForCausalLM, built from a Transformers Phi3Config converted from its remote-code config (it keeps going through the registry, so VLM2Vec still gets the embedding adapter). Kanana-V keeps the native LlamaForCausalLM: its remote-code text config can't be built by the Transformers modeling backend.

Gaps

• [Transformers backend] Support LoRA on lm_head and on any input embedding name #66413
  • LoRA adapters targeting lm_head are rejected
• [Transformers backend] Support EAGLE3 aux hidden states with pipeline parallelism and layer 0 #66546
  • Aux hidden state for layer 0 (extract_hidden_states, Eagle3) raises KeyError
  • Eagle3 with pipeline parallelism is rejected
  • Target embedding isn't kept on the last PP rank for drafters that share it
• [Transformers backend] Support batch-sharded sampling #66545
  • Batch-sharded sampling is silently disabled: no compute_logits_local
• [Transformers backend] Migrate ModernBERT embedding, reranker and token classification models #59206
  • LlamaModel/MistralModel embedding models without a sentence-transformers config default to CLS pooling instead of LAST
• [Bugfix] Fix KV scale names on stacked models and the --mm-encoder-only shared root check #65965
  • Quark checkpoints with {k,v}_proj.output_scale fail to load: the QKV fuser renames them before Quark's rule matches
• [Transformers backend] Fuse the GLU activation with the down projection's input quant #66809
  • NVFP4 is about 2% slower: the GLU fuser doesn't call maybe_fused_act_quant
• [BugFix] Undo llama.cpp's Q/K row permutation for Llama vllm-gguf-plugin#153, [BugFix] Give the tied GGUF lm_head a bias vllm-gguf-plugin#154 and a plugin release with [BugFix] Flatten batched inputs in the GGUF linear method vllm-gguf-plugin#149
  • GGUF fails to load: the plugin's linear assumes 2D activations (merged but unreleased) and the tied lm_head has no bias
  • Llama GGUF generates garbage on native and backend: llama.cpp's Q/K row permutation is never undone since [Migration] Migrate GGUF quantization support to plugin #39612
  • With all three: Llama-3.2-1B-Instruct Q8_0 GSM8K native 0.310, backend 0.308
• TODO
  • Warm start is about 6 s slower (25 s vs 19 s, 8B): the AOT compile artifact can't pickle HF's LlamaModel.forward
  • Existing tensorized Llama files stop loading: module names differ between implementations
  • Fusion e2e counts, perf dashboards and test_backend auto-vs-transformers comparisons assume native Llama
  • ROCm: AITER's rope + KV cache fusion doesn't fire because HF's rope isn't vLLM's rotary_embedding op, impact not measured

Validation

B200 bf16 Llama-3.1-8B-Instruct, native vs backend: GSM8K (500q, 5-shot) 0.782 vs 0.790, throughput (1024 in / 256 out) 43.15/43.24/43.15 vs 42.88/43.90/42.73 req/s, latency (bs 1) 0.956 vs 0.913 s. FP8 (compressed-tensors, ModelOpt), FP8 KV cache, W4A16, Quark FP8, GPTQ, AWQ, bitsandbytes, tensorizer, runai_streamer and TP=2 match native on GSM8K.

vllm bench throughput --model meta-llama/Llama-3.1-8B-Instruct --model-impl {vllm,transformers} --input-len 1024 --output-len 256 --num-prompts 1000

Passing on this branch: test_registry, test_config, test_common (Llama), and test_initialization for every Llama arch and native Llama dependent (Mistral, Llama 4, Eagle/Eagle3, GLM4, Jina v5, MiniCPM-V, Idefics3, LlamaBidirectional*) and every VLM whose language_model.model(...) call changed (50 passed after merging main; Eagle2.5 fails only because it's gated). InternLM3, IQuestCoder, TeleChat2/3 and IQuestLoopCoder pass test_initialization natively on this branch. Llava-1.5 generates correctly and passes multimodal test_common. Phi-3-vision-128k-instruct generates the same greedy text as main for an image and a text prompt.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

Documentation preview: https://vllm--66547.org.readthedocs.build/en/66547/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork