Skip to content

Fix LLMInferRequest to work with the input prompt of len 1 #32267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix LLMInferRequest to work with the input prompt of len 1 #32267

wants to merge 1 commit into from
+7 −3

Conversation

AsyaPronina
Copy link
Contributor

Details:

  • Fix LLM inference on NPU for input prompt of length 1

Tickets:

  • N/A

@AsyaPronina AsyaPronina requested review from a team as code owners October 1, 2025 13:43
@github-actions github-actions bot added category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin labels Oct 1, 2025
@dmatveev dmatveev added this to the 2025.4 milestone Oct 1, 2025
dmatveev
dmatveev reviewed Oct 1, 2025
View reviewed changes
src/plugins/intel_npu/src/plugin/npuw/llm_compiled_model.cpp
Comment on lines 545 to -546
uint32_t num_embeds_dim = 1 - batch_dim;
if (shape[num_embeds_dim] > max_generation_token_len) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somehow I overlooked it in the past but what is batch_dim? can this 1 - x underflow to some hugely positive value here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmatveev dmatveev dmatveev left review comments

@smirnov-alexey smirnov-alexey smirnov-alexey approved these changes

Assignees
No one assigned
Labels
category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin
Projects
None yet
Milestone
2025.4
Development

Successfully merging this pull request may close these issues.
3 participants
@AsyaPronina @dmatveev @smirnov-alexey