Repository navigation
Commit 6df6403
authored
[vLLM] Add Omni token counting and text replacement offsets for the Transformers backend (#49435)
* init: Add all files
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Split out the modeling-file changes
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* fix: Count Qwen2.5-Omni audio tokens from padded feature frames
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Build Gemma3n processor inputs through the ProcessorMixin hooks like Gemma 3
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Move the Gemma3n text type check into validate_inputs
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Move the Qwen-Omni processors onto the OmniModalProcessorMixin hooks
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* fix: Process audio as usual when load_audio_from_video gets no videos
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Address the round 3 review on the Qwen-Omni processors
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Address the round 4 review on the Qwen-Omni processors
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Address the round 5 review on the Omni processors
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* test: Move the audio token count test into the Mixin
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Address the round 7 review on the Omni processors
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* test: Seed the Cohere ASR double-shift test before the model init
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* fix: Count only standalone audio against the placeholders in the Omni mixin
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* fix: Return a list of prompts from the Gemma3n inputs layout for audio without text
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
---------
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>1 parent 85db0e0 commit 6df6403
13 files changed
Lines changed: 643 additions & 487 deletions
File tree
- src/transformers
- models
- embedding_gemma2
- gemma3n
- gemma4_unified
- gemma4
- qwen2_5_omni
- qwen3_omni_moe
- tests
- models
- cohere_asr
- gemma3n
- qwen2_5_omni
- qwen3_omni_moe
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
| 101 | + | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
104 | 133 | | |
105 | 134 | | |
106 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
259 | 259 | | |
260 | 260 | | |
261 | 261 | | |
262 | | - | |
263 | | - | |
| 262 | + | |
| 263 | + | |
264 | 264 | | |
265 | 265 | | |
266 | 266 | | |
| |||
288 | 288 | | |
289 | 289 | | |
290 | 290 | | |
291 | | - | |
292 | | - | |
| 291 | + | |
293 | 292 | | |
294 | 293 | | |
295 | 294 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments