Describe the feature request
We propose adding fusion transformers to reconstruct LM operators decomposed during WebNN EP graph processing: GroupQueryAttention, MatmulNBits, RMSNorm and GatherBlockQuantized.
These high-level operators are already present in the original ONNX model, but the WebNN EP decomposes them into primitive operations to meet WebNN API constraints. This decomposition affects both performance and numerical accuracy: native EPs such as WebGPU can no longer take advantage of their specialized kernels for these high-level operators.
Describe scenario use case
The target scenario is on-device LM inference through WebNN, using WebNN's ORT backend with native WebGPU EP. In phi4-mini demo, reconstructing these operators reduces graph complexity:
| Operator |
Node reduction |
| GroupQueryAttention |
About 3,030 GQA-related primitive nodes → 32 GQA nodes |
| MatMulNBits |
644 → 161 nodes |
| RMSNorm |
390 → 65 nodes |
| GatherBlockQuantized |
2 → 1 node, avoiding a roughly 1.145 GiB dequantized embedding table per invocation |
And these fusions improve decode IPOT from ~2000ms/token to ~60ms/token. The upstream path also exhibited accuracy issues; restoring RMSNorm with FP32 statistics resolved the generation correctness issue observed in our demo.
Describe the feature request
We propose adding fusion transformers to reconstruct LM operators decomposed during WebNN EP graph processing: GroupQueryAttention, MatmulNBits, RMSNorm and GatherBlockQuantized.
These high-level operators are already present in the original ONNX model, but the WebNN EP decomposes them into primitive operations to meet WebNN API constraints. This decomposition affects both performance and numerical accuracy: native EPs such as WebGPU can no longer take advantage of their specialized kernels for these high-level operators.
Describe scenario use case
The target scenario is on-device LM inference through WebNN, using WebNN's ORT backend with native WebGPU EP. In phi4-mini demo, reconstructing these operators reduces graph complexity:
And these fusions improve decode IPOT from ~2000ms/token to ~60ms/token. The upstream path also exhibited accuracy issues; restoring RMSNorm with FP32 statistics resolved the generation correctness issue observed in our demo.