-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable CPU helper in AUTO when the model is LLM #29233
base: master
Are you sure you want to change the base?
Disable CPU helper in AUTO when the model is LLM #29233
Conversation
…into guozhong/disable_cpu_helper
…into guozhong/disable_cpu_helper
b5d71a7
to
04212be
Compare
src/plugins/auto/src/plugin.cpp
Outdated
if (model_path.empty()) { | ||
support_devices = filter_device_by_model(support_devices_by_property, model, load_config); | ||
is_LLM_model = ov::op::util::is_large_language_model(*model); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it LLM specific issue or any model which has states?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ilya-lavrenov State model has been handle here
openvino/src/plugins/auto/src/plugin.cpp
Lines 906 to 919 in 5dba745
std::vector<std::string> stateful_node_names; | |
for (auto& op : model->get_ops()) { | |
if (ov::as_type_ptr<ov::op::util::AssignBase>(op) || | |
ov::as_type_ptr<ov::op::util::ReadValueBase>(op)) { | |
stateful_node_names.push_back(op->get_friendly_name()); | |
} | |
} | |
if (stateful_node_names.empty()) { | |
// not stateful model | |
return meta_devices; | |
} | |
// disable CPU_HELP and runtime fallback if model is stateful | |
disable_startup_runtime_fallback(); |
Here handle the LLM model only.
@wgzintel Why this is not handled in filter_device_by_model ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated and handled in filter_device_by_model.
src/plugins/auto/src/plugin.cpp
Outdated
if (is_LLM_model) { | ||
// disable cpu helper and runtime_fallback when the model is LLM, only one device need to compile model | ||
disable_startup_runtime_fallback(load_config); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider below code lines to replace the changed lines here.
auto m_model = model_path.empty() ? model : get_core()->read_model(mode_path, std::string{}, {});
and support_devices = filter_device_by_model(support_devices_by_property, m_model, load_config);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated
tickets: CVS-160732