Add Xiaomi-Lens-0 by guarin · Pull Request #50028 · huggingface/transformers

guarin · 2026-10-27T12:01:03Z

What does this PR do?

Fixes # (issue)

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. These often are low-quality, or fix extremely minor issues that occur rarely or never in practice.
As a result, we're instituting a rule that first-time contributors should not use code agents to submit PRs or issues.
We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues.

Issues/PRs from first-time contributors that violate this rule will probably just be closed without review, and we
might block you, especially if you open more than one or appear to be deliberately ignoring this. We especially do not
want new contributors to jump in on random issues to contribute an agent-written fix. This creates lots of noise
for reviewers and other users and will almost certainly get you blocked.

For more information, please read CONTRIBUTING.md.

(First-time contributors only): I confirm that this PR description and code is not written by an LLM or code agent

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline and the
Pull Request checks?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes according to the guidelines?
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

🤖 mlinter review state

hash: dac7099a
commit: 08c2fc23
review: Add Xiaomi-Lens-0 #50028 (review)

zucchini-nlp · 2026-10-28T03:21:48Z

+    image_features = model.get_image_features(
+        inputs.pop("pixel_values"), inputs.image_grid_thw, return_dict=True
+    )
+    generated = model.generate(
+        **inputs,
+        mm_encoder_outputs={"image": image_features},
+        return_dict_in_generate=True,
+        output_hidden_states=[model.config.text_config.num_hidden_layers - 1],
+        use_cache=True,
+        num_beams=1,
+        num_return_sequences=1,
+        max_new_tokens=1024,
+    )
+    hidden_states = torch.cat([step[-1] for step in generated.hidden_states], dim=1)
+    mask_logits, mask_token_indices = model.decode_masks(
+        generated.sequences, hidden_states, image_features.last_hidden_state,
+        inputs.image_grid_thw, ignore_thinking=True,
+    )


I had a very long plan to return image/video hidden states same way as we return encoder_hidden_states. It would be done auto-magically by routing model outputs directly into generation outputs so we dont have to manually infer/collect outputs per model

The person working on it left HF and the task got buried 😢

So the general plan i had was:

All multimodal models return same format mm_outputs, e.g. rn some models return only pooled images from VLM and some return a whole ModelOutput. IMO returning ModelOutput as in old blip models is the better way to allow flexibility

GenerationMixin split out the part of output collection into a smaller helper that mostly re-rereouts most fields returned from model to Mixin's output (we still dont return MoE router-logits and similar)

output = model.generate( **inputs, return_dict_in_generate=True, output_hidden_states=[model.config.text_config.num_hidden_layers - 1], max_new_tokens=1024, ) hidden_states = torch.cat([step[-1] for step in output.hidden_states], dim=1) image_hidden_states = output.image_hidden_states[-1] # it is already only last one returned in list mask_logits, mask_token_indices = model.decode_masks( output.sequences, hidden_states, image_hidden_states, inputs.image_grid_thw, ignore_thinking=True, )

…mi-lens

HuggingFaceDocBuilderDev · 2026-10-29T12:26:08Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

en/model_doc/xiaomi_lens0

molbap

hey, thanks! left an initial review, looks clean so far, not sure about the tracing. I like your proposal @zucchini-nlp , worth tackling IMO

molbap · 2026-10-30T09:33:24Z

+@require_vision
+class XiaomiLens0IntegrationTest(MemoryCleanupMixin, unittest.TestCase):
+    # TODO: Update before merge
+    model_id = "guarin/Xiaomi-Lens-0"  # "SeerRay-Lab/Xiaomi-Lens-0"


do we need a hub PR as well?

Yes, will open it once it is clear that no major changes are required

ok! let's ping on slack as well

molbap

few more generic comments

molbap · 2026-10-30T21:07:21Z

+        image_valid_mask = None if valid_targets_mask is None else valid_targets_mask[index]
+        num_masks += image_logits.shape[0] if image_valid_mask is None else image_valid_mask.flatten(1).any(-1).sum()
+
+        for start in range(0, image_logits.shape[0], chunk_size):


IIUC: this chunking only lowers memory under no grad (eval loss), in training autograd keeps every chunk's fp32 tensors until backward anyway. the original also checkpoints each chunk, that's where it saves, so maybe we can we drop the loop and mask_loss_chunk_size?

Fair, will drop it but then training will require a lot more compute.

ah really? I hadn't measured such a big loss tbh (just 7% more memory), but if you think it's worse we keep it as it was

molbap · 2026-10-30T21:07:21Z

+    elif labels is not None:
+        # Match the collective on ranks with mask supervision without connecting the decoder to the loss.
+        _normalize_mask_count(logits.new_zeros((), dtype=torch.float32))


not too familiar with the DDP side, if I follow correctly, a rank that only gets text batches never calls the decoder, so its params get no grad there. and with gradient checkpointing I think the Trainer sets find_unused_parameters=False, which I think crashes?

Exactly, it would only work with find_unused_parameters=True, also with other trainers than hf

ok so the docstring note is enough for me

molbap

Nice work 🚀
Approving for the "standard" stuff, for the custom generate maybe worth exploring an initial PR that does the returning of img hidden states cleanly and stack this one on top of it? I think it would be preferred against having a standalone class that does just one thing that is mostly a wrapper on utils we have

molbap · 2026-11-03T02:57:58Z

+    elif labels is not None:
+        # Match the collective on ranks with mask supervision without connecting the decoder to the loss.
+        _normalize_mask_count(logits.new_zeros((), dtype=torch.float32))


ok so the docstring note is enough for me

molbap · 2026-11-03T05:11:43Z

+@require_vision
+class XiaomiLens0IntegrationTest(MemoryCleanupMixin, unittest.TestCase):
+    # TODO: Update before merge
+    model_id = "guarin/Xiaomi-Lens-0"  # "SeerRay-Lab/Xiaomi-Lens-0"


ok! let's ping on slack as well

zucchini-nlp

adding my 2 cents 😄

btw, i will be making proper model output in multmodals. I got a local branch tho it still waits for an update for 40 existing models 😭
Left a comment re naming of model output tho to make refactor easier

zucchini-nlp · 2026-11-03T11:21:41Z

+        mask_logits = mask_embeddings @ pixel_embeddings.flatten(2) / self.config.mask_embed_dim**0.5
+        mask_logits = mask_logits.view(batch_size, -1, height, width)
+        # The fixed 4x upsampling brings the masks to the resolution of the image fed to the vision encoder
+        return F.interpolate(mask_logits, scale_factor=self.scale_factor, mode="bilinear", align_corners=False)


vision_utils.get_interpolte_indices for export friendliness? 😄

Hmm don't think that is needed here as it is just a simple interpolate and all logits have the same size

ah my bad, qwen ruined my life forever 😆

zucchini-nlp · 2026-11-12T01:16:15Z

+            return XiaomiLens0GenerateOutput(**result, image_hidden_states=image_hidden_states)
+        return result
+


image_last_hidden_state pls since it is only the last one returned

zucchini-nlp · 2026-11-12T01:17:23Z

+        Vision patch features before spatial merging, retaining gradients for mask supervision.
+    """
+
+    image_hidden_states: torch.FloatTensor | None = None


same here pls

github-actions · 2026-11-12T08:49:50Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, xiaomi_lens0

github-actions

Model linter — first pass

transformers-mlinter found 2 item(s) in the model files this PR touches. These are structural conventions a maintainer would otherwise flag by hand.

This is automated and advisory — it does not block merging.

rule	count	what it checks
`TRF055`	2	PreTrainedModel classes declare their config with a `config: SomeConfig` annotation, matched by `__init__(config: SomeConfig)`; never `config_class`.

github-actions · 2026-11-12T08:53:08Z

+    image_last_hidden_state: torch.Tensor | None = None
+
+
+class XiaomiLens0Model(Qwen3_5Model):


TRF055 — TRF055: generated modeling_xiaomi_lens0.py:76: XiaomiLens0Model.__init__ must annotate its config argument: config: XiaomiLens0Config, matching the config attribute it inherits or overrides.

Why this matters

PreTrainedModel derives config_class from the config annotation. An assignment or a hand-written config_class keeps or drifts from the parent's, and a mismatched __init__ builds the model from one config while it advertises another.

class Gemma4VisionModel(Gemma4PreTrainedModel): """The Gemma 4 Vision Encoder.""" - config = Gemma4VisionConfig + config: Gemma4VisionConfig - def __init__(self, config): + def __init__(self, config: Gemma4VisionConfig):

Suppress with # trf-ignore: TRF055 if intentional.

github-actions · 2026-11-12T08:53:08Z

+
+
+@auto_docstring
+class XiaomiLens0ForConditionalGeneration(Qwen3_5ForConditionalGeneration, XiaomiLens0GenerationMixin):


TRF055 — TRF055: generated modeling_xiaomi_lens0.py:734: XiaomiLens0ForConditionalGeneration.__init__ must annotate its config argument: config: XiaomiLens0Config, matching the config attribute it inherits or overrides.

Why this matters

PreTrainedModel derives config_class from the config annotation. An assignment or a hand-written config_class keeps or drifts from the parent's, and a mismatched __init__ builds the model from one config while it advertises another.

class Gemma4VisionModel(Gemma4PreTrainedModel): """The Gemma 4 Vision Encoder.""" - config = Gemma4VisionConfig + config: Gemma4VisionConfig - def __init__(self, config): + def __init__(self, config: Gemma4VisionConfig):

Suppress with # trf-ignore: TRF055 if intentional.

zucchini-nlp and others added 6 commits October 20, 2026 07:27

nit

d448060

seems no models are affected, deal with it later if someone complains

c18eb77

Add Xiaomi-Lens-0

3c159f5

update

de1b91a

restructure

ac3a944

update

4032a06

zucchini-nlp reviewed Oct 28, 2026

View reviewed changes

guarin added 12 commits October 28, 2026 11:50

add loss

57eecd2

update

ddd459a

Merge commit 'c18eb77e73781da762dc175670b1e4413f846045' into add-xiao…

592f7e7

…mi-lens

update

16d8dbd

update

d8c98dd

better hub compatibility

c9a593a

update tests

24df352

update tests

5b68805

update docs

7876580

fix repo

38edbf2

update docstrings

862fe63

fix repo

25eeec1

add loss test

4206c68

molbap reviewed Oct 30, 2026

View reviewed changes

guarin added 3 commits November 2, 2026 11:15

Merge branch 'main' into add-xiaomi-lens

bf18467

update

b04796a

update loss

9d29765

molbap approved these changes Nov 3, 2026

View reviewed changes

zucchini-nlp reviewed Nov 4, 2026

View reviewed changes

guarin added 2 commits November 9, 2026 11:36

Merge branch 'main' into add-xiaomi-lens

4ffb52a

rename to image_last_hidden_state

ffde6f9

guarin added 13 commits November 9, 2026 12:11

use AutoModel for vision/text

321aa18

Fix stuff

55d6bef

don't skip

3388e26

set target_sizes

530ddde

more cleanup

ee1aa28

Merge branch 'main' into add-xiaomi-lens

734ad20

fix repo

0e4fa2d

fix error

941b36a

move scale factor to config

acb47ff

cleanup

bb243ce

simplify

cae1859

better error messages

7a66032

simplify conversion mapping

08c2fc2

zucchini-nlp reviewed Nov 12, 2026

View reviewed changes

github-actions Bot reviewed Nov 12, 2026

View reviewed changes

		return XiaomiLens0GenerateOutput(**result, image_hidden_states=image_hidden_states)
		return result

		image_last_hidden_state: torch.Tensor \| None = None


		class XiaomiLens0Model(Qwen3_5Model):



		@auto_docstring
		class XiaomiLens0ForConditionalGeneration(Qwen3_5ForConditionalGeneration, XiaomiLens0GenerationMixin):

Conversation

guarin commented Oct 27, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guarin commented Oct 27, 2026 •

edited by github-actions Bot

Loading

HuggingFaceDocBuilderDev commented Oct 29, 2026 •

edited

Loading