Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable CPU helper in AUTO when the model is LLM #29233

Open
wants to merge 21 commits into
base: master
Choose a base branch
from

Conversation

wgzintel
Copy link
Contributor

@wgzintel wgzintel commented Mar 3, 2025

  1. Disable CPU helper in AUTO when the model is LLM.
  2. Fix the error "Not Implemented" when setting device to AUTO to run model InternVL2 on machines with more than two GPUs

tickets: CVS-160732

@github-actions github-actions bot added the category: AUTO OpenVINO AUTO device selection plugin label Mar 3, 2025
@github-actions github-actions bot added the category: transformations OpenVINO Runtime library - Transformations label Mar 3, 2025
@github-actions github-actions bot removed the category: transformations OpenVINO Runtime library - Transformations label Mar 4, 2025
@ilya-lavrenov ilya-lavrenov self-requested a review March 4, 2025 08:24
@wgzintel wgzintel force-pushed the guozhong/disable_cpu_helper branch from b5d71a7 to 04212be Compare March 5, 2025 15:10
@wgzintel wgzintel marked this pull request as ready for review March 10, 2025 01:53
@wgzintel wgzintel requested a review from a team as a code owner March 10, 2025 01:53
if (model_path.empty()) {
support_devices = filter_device_by_model(support_devices_by_property, model, load_config);
is_LLM_model = ov::op::util::is_large_language_model(*model);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it LLM specific issue or any model which has states?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ilya-lavrenov State model has been handle here

std::vector<std::string> stateful_node_names;
for (auto& op : model->get_ops()) {
if (ov::as_type_ptr<ov::op::util::AssignBase>(op) ||
ov::as_type_ptr<ov::op::util::ReadValueBase>(op)) {
stateful_node_names.push_back(op->get_friendly_name());
}
}
if (stateful_node_names.empty()) {
// not stateful model
return meta_devices;
}
// disable CPU_HELP and runtime fallback if model is stateful
disable_startup_runtime_fallback();
and it is being updated in https://github.com/openvinotoolkit/openvino/pull/27019/files#diff-85029a9232410831627d7b7b225e30d2cd4879e55b4abda5c475d04efb12daddL837-R859

Here handle the LLM model only.

@wgzintel Why this is not handled in filter_device_by_model ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and handled in filter_device_by_model.

if (is_LLM_model) {
// disable cpu helper and runtime_fallback when the model is LLM, only one device need to compile model
disable_startup_runtime_fallback(load_config);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider below code lines to replace the changed lines here.
auto m_model = model_path.empty() ? model : get_core()->read_model(mode_path, std::string{}, {}); and support_devices = filter_device_by_model(support_devices_by_property, m_model, load_config);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: AUTO OpenVINO AUTO device selection plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants