Repository navigation
Conversation
vLLM's Transformers modeling backend needs `_get_num_multimodal_tokens` to size placeholders without running the processor. Add it for image, video and audio processors whose models scatter features into `inputs_embeds`. The common image test now skips processors without an image processor, and the audio test's two-audios-per-text half can be disabled for processors that pair one audio with each text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
vLLM's Transformers backend only calls it with image and video sizes, so revert the audio-only processors and the audio branches of Inkling and Nemotron-H Omni. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
… overrides vLLM reads placeholders from `text_replacement_offsets`, not `mm_token_type_ids`, so revert the overrides and test the counts against the replacements instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…cation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Fills in the kwargs that weren't passed from the instance attributes and
standardizes them, replacing the hand-written defaulting in `preprocess`
and the `kwargs.get("k", self.k)` lookups in the token/patch counting
methods of image processors, video processors and processors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
`mm_token_type_ids` marks start/end/separator tokens for some models and not others, so compare `_get_num_multimodal_tokens` against the most repeated token in each item's `text_replacement_offsets` replacement instead, which is what vLLM counts. Align the helpers that also counted those extra tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…args Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, cohere_compass, colqwen2, cosmos3_edge, deepseek_ocr2, deepseek_vl, deepseek_vl_hybrid, dots3_note, embedding_gemma2, ernie4_5_vl_moe, exaone4_5, gemma4 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Note
Stacked on #50671. Only the last commit belongs to this PR; I'll rebase once #50671 merges.
Adds
BaseImageProcessor._merge_and_standardize_kwargs, which fills in the kwargs that weren't passed from the instance attributes and then standardizes them (as suggested in #50671 (comment)). Video processors inherit it.It's used wherever passed kwargs override the instance defaults:
preprocess: replaces the hand-written defaulting loop and_standardize_kwargscall in the image and video base classes, and in the three video processors that copied them (Ernie 4.5 VL MoE, MiniCPM-V 4.6, MiniCPM-OCR).get_number_of_image_patches/get_num_of_video_patches) now look upimages_kwargs["k"]instead ofimages_kwargs.get("k", self.k)/.get("k", None) or self.k._get_num_multimodal_tokensmethods (and LFM2-VL's_get_image_num_tokens) callself.image_processor._merge_and_standardize_kwargs(...)(or the video processor's) instead of reading defaults off it.The method name combines the existing
ProcessorMixin._merge_kwargs(defaults overridden by passed kwargs) and_standardize_kwargs.The conversion was done with an AST script that only rewrites a lookup when the key is one of the class's
valid_kwargsand the default is the attribute of the same name. Modular files were updated and the generated files regenerated.Behaviour changes
preprocess, e.g. a user-passedsizebecomes aSizeDict(Llama4's manual conversion is removed).None/0passed to a counting method is no longer replaced by the default (_merge_kwargsnever producesNonevalues, so this only affects manual calls).preprocessstandardizes kwargs before frame sampling rather than after.get_num_of_video_patchesmethods andMuseGlimmerImageProcessor.get_number_of_image_patchesno longer crash when called withoutvideos_kwargs/images_kwargs.Not converted: lookups whose key differs from the attribute (e.g.
get("patch_size", self.size)in GOT-OCR2 and Cohere2-Vision) and unrelatedkwargs.get(..., self.x)outside counting methods.Testing
pyteston everytest_processing_*,test_image_processing_*andtest_video_processing_*file of the 56 touched models (112 files): 3611 passed, 0 failed.test_image_processing_*andtest_video_processing_*files: 3967 passed, 1 failed (pp_doclayout_v3test_post_process, which also fails without this PR and isn't touched by it).utils/check_modular_conversion.pyandutils/check_copies.pyreport nothing for the touched files.This PR was written with AI assistance (Claude Code).
🤖 Generated with Claude Code