in paper 4.5, conv+bn is better in accuracy.
When I try to quantize model with low bit precision such as fp8 and int8, conv+bn can be fused in one opt, but conv+ln can not and usually need some extra opts such as permute, transpose, which makes vit arch models in fp8 and int8 mode are slower than fp16 mode
so, are bn verison model checkpoints still there? can they be published?
finally and personally, it would be better to consider using low bit quantize model to compare performance and accuracy in the future.
thank you for your arts, hopping any reply :)
in paper 4.5,
conv+bnis better in accuracy.When I try to quantize model with low bit precision such as
fp8andint8,conv+bncan be fused in one opt, butconv+lncan not and usually need some extra opts such aspermute,transpose, which makes vit arch models infp8andint8mode are slower thanfp16modeso, are
bnverison model checkpoints still there? can they be published?finally and personally, it would be better to consider using low bit quantize model to compare performance and accuracy in the future.
thank you for your arts, hopping any reply :)