Skip to content

Add _merge_and_standardize_kwargs to image and video processors - #50706

Draft
hmellor wants to merge 10 commits into
huggingface:mainfrom
hmellor:merge-and-standardize-kwargs
Draft

hmellor wants to merge 10 commits into
huggingface:mainfrom
hmellor:merge-and-standardize-kwargs

Conversation

@hmellor

@hmellor hmellor commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

🤗 Transformers CI

What does this PR do?

Note

Stacked on #50671. Only the last commit belongs to this PR; I'll rebase once #50671 merges.

Adds BaseImageProcessor._merge_and_standardize_kwargs, which fills in the kwargs that weren't passed from the instance attributes and then standardizes them (as suggested in #50671 (comment)). Video processors inherit it.

It's used wherever passed kwargs override the instance defaults:

  • preprocess: replaces the hand-written defaulting loop and _standardize_kwargs call in the image and video base classes, and in the three video processors that copied them (Ernie 4.5 VL MoE, MiniCPM-V 4.6, MiniCPM-OCR).
  • Image/video processor counting methods: 82 methods (mostly get_number_of_image_patches / get_num_of_video_patches) now look up images_kwargs["k"] instead of images_kwargs.get("k", self.k) / .get("k", None) or self.k.
  • Processors: ~40 _get_num_multimodal_tokens methods (and LFM2-VL's _get_image_num_tokens) call self.image_processor._merge_and_standardize_kwargs(...) (or the video processor's) instead of reading defaults off it.

The method name combines the existing ProcessorMixin._merge_kwargs (defaults overridden by passed kwargs) and _standardize_kwargs.

The conversion was done with an AST script that only rewrites a lookup when the key is one of the class's valid_kwargs and the default is the attribute of the same name. Modular files were updated and the generated files regenerated.

Behaviour changes

  • Passed kwargs in the counting methods are now standardized like in preprocess, e.g. a user-passed size becomes a SizeDict (Llama4's manual conversion is removed).
  • An explicit None/0 passed to a counting method is no longer replaced by the default (_merge_kwargs never produces None values, so this only affects manual calls).
  • Video preprocess standardizes kwargs before frame sampling rather than after.
  • 13 get_num_of_video_patches methods and MuseGlimmerImageProcessor.get_number_of_image_patches no longer crash when called without videos_kwargs/images_kwargs.

Not converted: lookups whose key differs from the attribute (e.g. get("patch_size", self.size) in GOT-OCR2 and Cohere2-Vision) and unrelated kwargs.get(..., self.x) outside counting methods.

Testing

  • pytest on every test_processing_*, test_image_processing_* and test_video_processing_* file of the 56 touched models (112 files): 3611 passed, 0 failed.
  • All test_image_processing_* and test_video_processing_* files: 3967 passed, 1 failed (pp_doclayout_v3 test_post_process, which also fails without this PR and isn't touched by it).
  • utils/check_modular_conversion.py and utils/check_copies.py report nothing for the touched files.

This PR was written with AI assistance (Claude Code).

🤖 Generated with Claude Code

hmellor and others added 8 commits November 12, 2026 17:13
vLLM's Transformers modeling backend needs `_get_num_multimodal_tokens` to
size placeholders without running the processor. Add it for image, video
and audio processors whose models scatter features into `inputs_embeds`.

The common image test now skips processors without an image processor, and
the audio test's two-audios-per-text half can be disabled for processors
that pair one audio with each text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
vLLM's Transformers backend only calls it with image and video sizes, so
revert the audio-only processors and the audio branches of Inkling and
Nemotron-H Omni.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
… overrides

vLLM reads placeholders from `text_replacement_offsets`, not
`mm_token_type_ids`, so revert the overrides and test the counts against
the replacements instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…cation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Fills in the kwargs that weren't passed from the instance attributes and
standardizes them, replacing the hand-written defaulting in `preprocess`
and the `kwargs.get("k", self.k)` lookups in the token/patch counting
methods of image processors, video processors and processors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hmellor and others added 2 commits November 13, 2026 16:29
`mm_token_type_ids` marks start/end/separator tokens for some models and not
others, so compare `_get_num_multimodal_tokens` against the most repeated token
in each item's `text_replacement_offsets` replacement instead, which is what
vLLM counts. Align the helpers that also counted those extra tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…args

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, cohere_compass, colqwen2, cosmos3_edge, deepseek_ocr2, deepseek_vl, deepseek_vl_hybrid, dots3_note, embedding_gemma2, ernie4_5_vl_moe, exaone4_5, gemma4

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants