Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing constant propagation: Literal -> Multibroadcast -> Quantizelinear #3597

Open
CharlieL7 opened this issue Nov 7, 2024 · 0 comments
Open
Labels
FP8 issues related to FP8 implemenation INT8 Perf Improve

Comments

@CharlieL7
Copy link
Collaborator

CharlieL7 commented Nov 7, 2024

  • Found during Inference Model Review meeting
  • Seen in bert_base_cased and distilgpt2_fp16 run with our --fp8 flag and probably also --int8
@12 = hip::hip_copy_literal[id=main:@literal:17] -> half_type, {768, 2304}, {2304, 1}
@13 = load[offset=188743680,end=190513152](@1) -> fp8e4m3fnuz_type, {64, 768, 2304}, {0, 2304, 1}
@14 = multibroadcast[out_lens={64, 768, 2304},out_dyn_dims={}](@12) -> half_type, {64, 768, 2304}, {0, 2304, 1}
@15 = gpu::code_object[code_object=5088,symbol_name=quantizelinear_kernel,global=113246208,local=1024,](@14,@13) -> fp8e4m3fnuz_type, {64, 768, 2304}, {0, 2304, 1}
  • Example from distilgpt2_fp16
    • driver command: bin/driver perf /codes/distilgpt2_1_fp16_gpu.onnx --fp8 --fill1 input_ids --input-dim @input_ids 64 384 --batch 64
  • The input to instruction @15 quantizelinear is a broadcasted literal. The broadcast instruction should have been swapped by the find_inner_broadcasts matcher in the simplify_algebra compiler pass to then allow the propagate_constant pass to make it into a constant.
@CharlieL7 CharlieL7 added FP8 issues related to FP8 implemenation Perf Improve INT8 labels Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FP8 issues related to FP8 implemenation INT8 Perf Improve
Projects
None yet
Development

No branches or pull requests

1 participant