Add Xiaomi-Lens-0

docs/source/en/_toctree.yml

@@ -1605,6 +1605,8 @@
         title: VoxtralRealtime
       - local: model_doc/xclip
         title: X-CLIP
+      - local: model_doc/xiaomi_lens0
+        title: Xiaomi-Lens-0
       title: Multimodal models
     - sections:
       - local: model_doc/decision_transformer

docs/source/en/model_doc/xiaomi_lens0.md

@@ -0,0 +1,154 @@
+<!--Copyright 2026 the HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
+⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.
+
+-->
+*This model was contributed to Hugging Face Transformers on 2026-10-29.*
+
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+        <img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
+        <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+    </div>
+</div>
+
+# Xiaomi-Lens-0
+
+## Overview
+
+[Xiaomi-Lens-0](https://huggingface.co/papers/2610.23279) is a 4B multimodal language model built on [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). It adds a lightweight pixel-mask decoder and is trained with joint language and mask supervision. Each generated `<|mask|>` token identifies an image region that the decoder turns into a segmentation mask.
+
+The model supports referring expression comprehension and segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained visual perception. It can produce masks in its final answer or during reasoning. See the [model card](https://huggingface.co/SeerRay-Lab/Xiaomi-Lens-0) for all examples and details.
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xiaomi_lens0_architecture.jpg"
+alt="xiaomi lens0 architecture overview" width="600"/>
+
+The original code can be found [here](https://github.com/SeerRay-Lab/Xiaomi-Lens-0).
+
+<hfoptions id="usage">
+<hfoption id="AutoModelForMultimodalLM">
+
+```python
+import torch
+from transformers import AutoModelForMultimodalLM, AutoProcessor
+
+model_id = "SeerRay-Lab/Xiaomi-Lens-0"
+model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
+            {
+                "type": "text",
+                "text": 'Locate all instances in the image that match the description "cat".\nReport bbox coordinates and masks.',
+            },
+        ],
+    }
+]
+inputs = processor.apply_chat_template(
+    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
+).to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        return_dict_in_generate=True,
+        output_hidden_states=[model.config.text_config.num_hidden_layers - 1],
+        max_new_tokens=1024,
+    )
+    hidden_states = torch.cat([step[-1] for step in outputs.hidden_states], dim=1)
+    mask_logits, mask_token_indices = model.decode_masks(
+        input_ids=outputs.sequences,
+        hidden_states=hidden_states,
+        image_hidden_states=outputs.image_hidden_states,
+        image_grid_thw=inputs.image_grid_thw,
+    )
+results = processor.post_process_grounding(
+    outputs.sequences, inputs.image_grid_thw, mask_logits, mask_token_indices, target_sizes=[(480, 640)]
+)
+print(results[0]["boxes"]) # bounding boxes in absolute (top_left_x, top_left_y, bottom_right_x, bottom_right_y) pixel coordinates
+# tensor([[344.9600,  24.9600, 640.0000, 372.0000],
+#         [  8.9600,  55.2000, 316.1600, 474.2400]])
+print(results[0]["masks"].shape) # binary segmentation masks
+# torch.Size([2, 480, 640])
+```
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xiaomi_lens0_grounding.png"
+alt="xiaomi lens0 grounding boxes and masks results" width="600"/>
+
+</hfoption>
+</hfoptions>
+
+## Notes
+
+- Boxes are written as `<bbox>[x1, y1, x2, y2]</bbox>` with coordinates normalized to `[0, 1000]`, and masks as `<mask><|mask|></mask>`. A grounded instance is written as `<bbox>[x1, y1, x2, y2]</bbox><mask><|mask|></mask>`.
+- Ask for boxes and masks explicitly with "Report bbox coordinates and masks." at the end of the instruction.
+- To detect boxes or masks only ask with "Report bbox coordinates." or "Report masks."
+- The mask decoder sees the image at the resolution chosen by the image processor (up to `max_pixels`). Masks are predicted at that resolution and [`~XiaomiLens0Processor.post_process_grounding`] resizes them to `target_sizes`.
+- Decoding and postprocessing exclude boxes and masks from thinking blocks by default. Set `ignore_thinking=False` in [`~XiaomiLens0ForConditionalGeneration.decode_masks`] to include masks and in [`~XiaomiLens0Processor.post_process_grounding`] to include boxes.
+
+## XiaomiLens0Config
+
+[[autodoc]] XiaomiLens0Config
+
+## XiaomiLens0TextConfig
+
+[[autodoc]] XiaomiLens0TextConfig
+
+## XiaomiLens0VisionConfig
+
+[[autodoc]] XiaomiLens0VisionConfig
+
+## XiaomiLens0MaskDecoderConfig
+
+[[autodoc]] XiaomiLens0MaskDecoderConfig
+
+## XiaomiLens0Processor
+
+[[autodoc]] XiaomiLens0Processor
+    - post_process_grounding
+
+## XiaomiLens0PreTrainedModel
+
+[[autodoc]] XiaomiLens0PreTrainedModel
+
+## XiaomiLens0VisionModel
+
+[[autodoc]] XiaomiLens0VisionModel
+    - forward
+
+## XiaomiLens0TextModel
+
+[[autodoc]] XiaomiLens0TextModel
+    - forward
+
+## XiaomiLens0Model
+
+[[autodoc]] XiaomiLens0Model
+    - forward
+
+## XiaomiLens0ForConditionalGeneration
+
+[[autodoc]] XiaomiLens0ForConditionalGeneration
+    - forward
+    - decode_masks
+
+## XiaomiLens0GenerateOutput
+
+[[autodoc]] XiaomiLens0GenerateOutput

src/transformers/conversion_mapping.py

@@ -1149,6 +1149,51 @@ def _build_checkpoint_conversion_mapping():
             ),
         ],
         "qwen3_5_text": [PrefixChange(prefix_to_remove="language_model", model_prefix="model")],
+        # Xiaomi-Lens-0 mask decoder: unroll the original `nn.Sequential` blocks and split the fused
+        # `nn.MultiheadAttention` input projection
+        "xiaomi_lens0": [
+            WeightRenaming(r"mask_decoder\.mask_embed\.net\.0\.", "mask_decoder.mask_embed.fc1."),
+            WeightRenaming(r"mask_decoder\.mask_embed\.net\.2\.", "mask_decoder.mask_embed.fc2."),
+            WeightRenaming(r"mask_decoder\.patch_proj\.", "mask_decoder.pixel_decoder.patch_proj."),
+            WeightRenaming(
+                r"mask_decoder\.patch_self_attn\.norm1\.", "mask_decoder.pixel_decoder.patch_layer.layer_norm1."
+            ),
+            WeightRenaming(
+                r"mask_decoder\.patch_self_attn\.norm2\.", "mask_decoder.pixel_decoder.patch_layer.layer_norm2."
+            ),
+            WeightRenaming(
+                r"mask_decoder\.patch_self_attn\.mlp\.net\.0\.", "mask_decoder.pixel_decoder.patch_layer.mlp.fc1."
+            ),
+            WeightRenaming(
+                r"mask_decoder\.patch_self_attn\.mlp\.net\.2\.", "mask_decoder.pixel_decoder.patch_layer.mlp.fc2."
+            ),
+            WeightRenaming(
+                r"mask_decoder\.patch_self_attn\.attn\.out_proj\.",
+                "mask_decoder.pixel_decoder.patch_layer.self_attn.out_proj.",
+            ),
+            WeightConverter(
+                source_patterns="mask_decoder.patch_self_attn.attn.in_proj_weight",
+                target_patterns=[
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.q_proj.weight",
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.k_proj.weight",
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.v_proj.weight",
+                ],
+                operations=[Chunk(dim=0)],
+            ),
+            WeightConverter(
+                source_patterns="mask_decoder.patch_self_attn.attn.in_proj_bias",
+                target_patterns=[
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.q_proj.bias",
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.k_proj.bias",
+                    "mask_decoder.pixel_decoder.patch_layer.self_attn.v_proj.bias",
+                ],
+                operations=[Chunk(dim=0)],
+            ),
+            WeightRenaming(r"mask_decoder\.channel_reduce\.", "mask_decoder.pixel_decoder.channel_reduce."),
+            WeightRenaming(r"mask_decoder\.upscaler\.0\.", "mask_decoder.pixel_decoder.upscale_block.conv1."),
+            WeightRenaming(r"mask_decoder\.upscaler\.1\.", "mask_decoder.pixel_decoder.upscale_block.layer_norm."),
+            WeightRenaming(r"mask_decoder\.upscaler\.3\.", "mask_decoder.pixel_decoder.upscale_block.conv2."),
+        ],
         "sam3_tracker": [
             WeightRenaming(
                 source_patterns=r"detector_model.vision_encoder.backbone.", target_patterns="vision_encoder.backbone."

src/transformers/loss/loss_utils.py

@@ -29,6 +29,7 @@
 from .loss_rnnt import ParakeetForRNNTLoss
 from .loss_rt_detr import RTDetrForObjectDetectionLoss
 from .loss_tdt import ParakeetForTDTLoss
+from .loss_xiaomi_lens0 import XiaomiLens0ForConditionalGenerationLoss
 
 
 def fixed_cross_entropy(
@@ -203,4 +204,5 @@ def ForSemanticSegmentationLoss(
     "ParakeetForTDT": ParakeetForTDTLoss,
     "RfDetrForObjectDetection": LwDetrForObjectDetectionLoss,
     "RfDetrForInstanceSegmentation": RfDetrForSegmentationLoss,
+    "XiaomiLens0ForConditionalGeneration": XiaomiLens0ForConditionalGenerationLoss,
 }

src/transformers/loss/loss_xiaomi_lens0.py

@@ -0,0 +1,168 @@
+# Copyright 2026 Xiaomi Corporation and The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.distributed as dist
+import torch.nn.functional as F
+
+
+def _normalize_mask_count(num_masks: torch.Tensor) -> torch.Tensor:
+    if dist.is_available() and dist.is_initialized():
+        dist.all_reduce(num_masks)
+        num_masks = num_masks / dist.get_world_size()
+    return num_masks.clamp_min(1.0)
+
+
+def xiaomi_lens0_mask_loss(
+    logits: list[torch.Tensor],
+    targets: list[torch.Tensor],
+    valid_targets_mask: list[torch.Tensor] | None,
+    dice_weight: float,
+    bce_weight: float,
+    chunk_size: int,
+) -> torch.Tensor:
+    r"""
+    Compute the weighted Dice and sigmoid binary cross-entropy loss over valid masks.
+
+    Args:
+        logits (`list[torch.Tensor]`):
+            Non-empty list of raw mask logits in input-image order. Each tensor has shape
+            `(num_masks, height, width)`; mask counts and spatial dimensions may differ between images.
+            Images without mask tokens can have tensors with `num_masks=0`.
+        targets (`list[torch.Tensor]`):
+            Binary segmentation targets with the same list length and tensor shapes as `logits`.
+            Masks within each image must follow the same order as the logits.
+        valid_targets_mask (`list[torch.Tensor]` or `None`):
+            Boolean or binary tensors matching `logits`, with 1 indicating a supervised pixel and 0 an
+            ignored pixel. Masks with no valid pixels are excluded from the mask count. If `None`, all pixels
+            and masks are included.
+        dice_weight (`float`):
+            Weight of the Dice loss computed from sigmoid probabilities, with a smoothing constant of 1.
+        bce_weight (`float`):
+            Weight of the binary cross-entropy loss, averaged over valid pixels within each mask.
+        chunk_size (`int`):
+            Number of masks processed per chunk for each image. Logits and targets are converted to
+            float per chunk to limit memory usage.
+
+    Returns:
+        `torch.Tensor` of shape `()`: Scalar float32 loss. Weighted per-mask losses are summed locally and divided
+        by the valid-mask count, clamped to at least 1. When distributed training is initialized, the divisor is
+        the global valid-mask count divided by the world size, also clamped to at least 1.
+    """
+    total_loss = 0
+    num_masks = logits[0].new_zeros((), dtype=torch.float32)
+    for index, (image_logits, image_targets) in enumerate(zip(logits, targets)):
+        image_valid_mask = None if valid_targets_mask is None else valid_targets_mask[index]
+        num_masks += image_logits.shape[0] if image_valid_mask is None else image_valid_mask.flatten(1).any(-1).sum()
+
+        for start in range(0, image_logits.shape[0], chunk_size):
+            chunk_logits = image_logits[start : start + chunk_size].float().flatten(1)
+            chunk_targets = image_targets[start : start + chunk_size].float().flatten(1)
+            probabilities = chunk_logits.sigmoid()
+
+            if image_valid_mask is None:
+                bce_loss = (
+                    F.binary_cross_entropy_with_logits(chunk_logits, chunk_targets, reduction="mean")
+                    * chunk_logits.shape[0]
+                )
+            else:
+                chunk_valid_mask = image_valid_mask[start : start + chunk_size].float().flatten(1)
+                weights = chunk_valid_mask / chunk_valid_mask.sum(-1, keepdim=True).clamp_min(1.0)
+                bce_loss = F.binary_cross_entropy_with_logits(
+                    chunk_logits, chunk_targets, weight=weights, reduction="sum"
+                )
+                probabilities = probabilities * chunk_valid_mask
+                chunk_targets = chunk_targets * chunk_valid_mask
+
+            intersection = 2 * (probabilities * chunk_targets).sum(-1)
+            denominator = probabilities.sum(-1) + chunk_targets.sum(-1)
+            dice_loss = (1 - (intersection + 1) / (denominator + 1)).sum()
+            total_loss = total_loss + dice_weight * dice_loss + bce_weight * bce_loss
+
+    return total_loss / _normalize_mask_count(num_masks)
+
+
+def XiaomiLens0ForConditionalGenerationLoss(
+    logits: torch.Tensor,
+    labels: torch.Tensor | None,
+    mask_logits: list[torch.Tensor] | None,
+    mask_labels: list[torch.Tensor] | None,
+    mask_labels_valid_mask: list[torch.Tensor] | None,
+    config,
+    **kwargs,
+) -> tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]:
+    r"""
+    Compute the language modeling loss and weighted mask loss for Xiaomi-Lens-0.
+
+    Every supervised call participates in mask-count normalization, including language-only batches, which
+    contribute zero masks without running the mask decoder. In distributed training, all ranks must call this
+    loss with language labels or mask labels. Distributed wrappers must support unused parameters when a rank
+    has no mask supervision.
+
+    Args:
+        logits (`torch.Tensor` of shape `(batch_size, sequence_length, vocab_size)`):
+            Raw language-model logits. Also determines the device for mask-count normalization when mask labels
+            are absent.
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)` or `None`):
+            Language-model targets. Labels are shifted internally for next-token prediction, and values of -100
+            are ignored by default. Pass `None` to omit the language modeling loss.
+        mask_logits (`list[torch.Tensor]` or `None`):
+            Raw mask logits grouped in input-image order, each of shape `(num_masks, height, width)`.
+            Required when `mask_labels` is provided; otherwise ignored.
+        mask_labels (`list[torch.Tensor]` or `None`):
+            Binary segmentation targets matching the list length, shapes, and mask order of `mask_logits`.
+            Pass `None` to omit the mask loss. Individual image tensors may contain zero masks.
+        mask_labels_valid_mask (`list[torch.Tensor]` or `None`):
+            Boolean or binary pixel-validity tensors matching `mask_labels`. Zero-valued pixels are ignored;
+            masks with no valid pixels are excluded from normalization. Pass `None` to supervise all pixels.
+            Only used when `mask_labels` is provided.
+        config ([`XiaomiLens0Config`]):
+            Model configuration supplying `text_config.vocab_size`, `mask_loss_weight`,
+            `mask_dice_loss_weight`, `mask_bce_loss_weight`, and `mask_loss_chunk_size`.
+        **kwargs:
+            Additional arguments forwarded to `ForCausalLMLoss`, such as `num_items_in_batch`, `ignore_index`,
+            and pre-shifted `shift_labels`.
+
+    Returns:
+        `tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]`: A tuple containing:
+
+        - **loss** (`torch.Tensor` of shape `()` or `None`) -- Sum of the language modeling loss and weighted
+          mask loss. If only one component is supervised, returns that component. Returns `None` if both
+          `labels` and `mask_labels` are `None`.
+        - **lm_loss** (`torch.Tensor` of shape `()` or `None`) -- Language modeling loss returned by
+          `ForCausalLMLoss`, or `None` when `labels` is absent.
+        - **mask_loss** (`torch.Tensor` of shape `()` or `None`) -- Dice and binary cross-entropy loss
+          multiplied by `config.mask_loss_weight`, or `None` when `mask_labels` is absent.
+    """
+    # Import here because loss_utils registers this function in LOSS_MAPPING.
+    from .loss_utils import ForCausalLMLoss
+
+    loss = lm_loss = mask_loss = None
+    if labels is not None:
+        lm_loss = ForCausalLMLoss(logits=logits, labels=labels, vocab_size=config.text_config.vocab_size, **kwargs)
+        loss = lm_loss
+
+    if mask_labels is not None:
+        mask_loss = config.mask_loss_weight * xiaomi_lens0_mask_loss(
+            mask_logits,
+            mask_labels,
+            valid_targets_mask=mask_labels_valid_mask,
+            dice_weight=config.mask_dice_loss_weight,
+            bce_weight=config.mask_bce_loss_weight,
+            chunk_size=config.mask_loss_chunk_size,
+        )
+        loss = mask_loss if loss is None else loss + mask_loss
+    elif labels is not None:
+        # Match the collective on ranks with mask supervision without connecting the decoder to the loss.
+        _normalize_mask_count(logits.new_zeros((), dtype=torch.float32))
+    return loss, lm_loss, mask_loss