|
| 1 | +<!--Copyright 2026 the HuggingFace Team. All rights reserved. |
| 2 | +
|
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | +you may not use this file except in compliance with the License. |
| 5 | +You may obtain a copy of the License at |
| 6 | +
|
| 7 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +Unless required by applicable law or agreed to in writing, software |
| 10 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | +See the License for the specific language governing permissions and |
| 13 | +limitations under the License. |
| 14 | +
|
| 15 | +⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. |
| 16 | +
|
| 17 | +--> |
| 18 | +*This model was contributed to Hugging Face Transformers on 2026-10-29.* |
| 19 | + |
| 20 | +<div style="float: right;"> |
| 21 | + <div class="flex flex-wrap space-x-1"> |
| 22 | + <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white"> |
| 23 | + </div> |
| 24 | +</div> |
| 25 | + |
| 26 | +# Mistral Large 4 |
| 27 | + |
| 28 | +## Overview |
| 29 | + |
| 30 | +Mistral Large 4 is a natively multimodal mixture-of-experts model with 1 trillion parameters and 49 billion active |
| 31 | +parameters per token (52 billion including the embeddings and the output layer). It takes text and images as input, |
| 32 | +produces text, and can answer directly or reason first, with the reasoning effort set per request. |
| 33 | + |
| 34 | +The text decoder builds on the DeepSeek-V3.2 recipe (Multi-head Latent Attention made sparse by a DSA indexer) and adds |
| 35 | +a few changes of its own: |
| 36 | + |
| 37 | +- **Attention**: 72 layers of Deepseek Sparse Attention (DSA) with a 512-dimensional KV latent, made sparse by an |
| 38 | + indexer selecting the top 2048 key positions for each query. Queries use a direct projection, there is no Q-LoRA. |
| 39 | + Some layers compute their own selection, the others reuse the one of a previous layer (`"shared"`) like in |
| 40 | + [GLM-5](./glm_moe_dsa). |
| 41 | +- **NoPE layers**: every 4th layer uses no rotary embedding, in the attention and in the indexer. |
| 42 | +- **KV sharing**: from layer 39 and on, every 4th layer does not compute a KV latent, instead attending to the one of |
| 43 | + layer 35. They keep their own query, KV up-projection and output weights. |
| 44 | +- **Scaled softmax**: queries are scaled per head by `s * log(position + 1) + b`, with learned `s` and `b`, in the |
| 45 | + attention and in the indexer. |
| 46 | +- **LeRoPE**: YaRN extends the 512-token base RoPE by a factor of 2048 (1,048,576 positions), and LeRoPE adds a |
| 47 | + learned multiplier per rotary frequency. |
| 48 | +- **MoE**: the first layer is dense, every other layer routes each token to 4 of 128 experts, plus one shared expert. |
| 49 | +- **Logits**: final logits are soft-capped with `30 * tanh(logits / 30)`. |
| 50 | + |
| 51 | +The vision encoder and the multimodal projector are the same as in [Mistral 3](./mistral3) (Pixtral encoder with a |
| 52 | +patch-merge projector). |
| 53 | + |
| 54 | +## Usage examples |
| 55 | + |
| 56 | +The checkpoint is stored in FP8 (128x128 block-scaled weights with dynamic activation scales) and is loaded as is. |
| 57 | +The full model needs about 1 TB of GPU memory, so `device_map="auto"` is used to spread it across several GPUs. |
| 58 | + |
| 59 | +The chat template accepts a `reasoning_effort` argument, set to `"none"` (default) or `"high"`. With `"high"`, the model |
| 60 | +writes its reasoning between `[THINK]` and `[/THINK]` before answering. |
| 61 | + |
| 62 | +<hfoptions id="usage"> |
| 63 | +<hfoption id="Image and text"> |
| 64 | + |
| 65 | +```python |
| 66 | +from transformers import AutoProcessor, MistralLarge4ForConditionalGeneration |
| 67 | + |
| 68 | +model_id = "mistralai/Mistral-Large-4-1T-A52B" |
| 69 | + |
| 70 | +processor = AutoProcessor.from_pretrained(model_id) |
| 71 | +model = MistralLarge4ForConditionalGeneration.from_pretrained(model_id, device_map="auto") |
| 72 | + |
| 73 | +messages = [ |
| 74 | + { |
| 75 | + "role": "user", |
| 76 | + "content": [ |
| 77 | + {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"}, |
| 78 | + {"type": "text", "text": "Describe this image."}, |
| 79 | + ], |
| 80 | + }, |
| 81 | +] |
| 82 | + |
| 83 | +inputs = processor.apply_chat_template( |
| 84 | + messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" |
| 85 | +).to(model.device) |
| 86 | + |
| 87 | +output = model.generate(**inputs, max_new_tokens=256) |
| 88 | +print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) |
| 89 | +``` |
| 90 | + |
| 91 | +</hfoption> |
| 92 | +<hfoption id="Reasoning"> |
| 93 | + |
| 94 | +```python |
| 95 | +from transformers import AutoProcessor, MistralLarge4ForConditionalGeneration |
| 96 | + |
| 97 | +model_id = "mistralai/Mistral-Large-4-1T-A52B" |
| 98 | + |
| 99 | +processor = AutoProcessor.from_pretrained(model_id) |
| 100 | +model = MistralLarge4ForConditionalGeneration.from_pretrained(model_id, device_map="auto") |
| 101 | + |
| 102 | +messages = [{"role": "user", "content": "How many prime numbers are there between 100 and 150?"}] |
| 103 | + |
| 104 | +inputs = processor.apply_chat_template( |
| 105 | + messages, |
| 106 | + add_generation_prompt=True, |
| 107 | + tokenize=True, |
| 108 | + return_dict=True, |
| 109 | + return_tensors="pt", |
| 110 | + reasoning_effort="high", |
| 111 | +).to(model.device) |
| 112 | + |
| 113 | +output = model.generate(**inputs, max_new_tokens=4096) |
| 114 | +# Keep the special tokens to see the reasoning between [THINK] and [/THINK] |
| 115 | +print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False)) |
| 116 | +``` |
| 117 | + |
| 118 | +</hfoption> |
| 119 | +</hfoptions> |
| 120 | + |
| 121 | +## Notes |
| 122 | + |
| 123 | +- The checkpoint also contains a draft head for speculative decoding (the `mtp_dspark.*` weights) using the DSpark |
| 124 | + speculative decoding method. <!-- TODO: implement DSpark or specify this is not supported --> |
| 125 | +- The DSA indexer is applied as an attention mask on top of dense attention, with the `eager` and `sdpa` backends. |
| 126 | + It gives the same outputs as sparse attention but not its speed-up on long sequences. |
| 127 | +<!-- |
| 128 | +TODO: ask tokenizer people about this |
| 129 | +- The tokenizer reads a literal `[INST]` typed in a user message as the control token, while the reference |
| 130 | + [mistral-common](https://github.com/mistralai/mistral-common) tokenizer keeps it as plain text. --> |
| 131 | + |
| 132 | +## MistralLarge4Config |
| 133 | + |
| 134 | +[[autodoc]] MistralLarge4Config |
| 135 | + |
| 136 | +## MistralLarge4TextConfig |
| 137 | + |
| 138 | +[[autodoc]] MistralLarge4TextConfig |
| 139 | + |
| 140 | +## MistralLarge4PreTrainedModel |
| 141 | + |
| 142 | +[[autodoc]] MistralLarge4PreTrainedModel |
| 143 | + |
| 144 | +## MistralLarge4TextModel |
| 145 | + |
| 146 | +[[autodoc]] MistralLarge4TextModel |
| 147 | + - forward |
| 148 | + |
| 149 | +## MistralLarge4Model |
| 150 | + |
| 151 | +[[autodoc]] MistralLarge4Model |
| 152 | + - forward |
| 153 | + |
| 154 | +## MistralLarge4ForConditionalGeneration |
| 155 | + |
| 156 | +[[autodoc]] MistralLarge4ForConditionalGeneration |
| 157 | + - forward |
0 commit comments