[Mobile] [QNN EP GPU] Noisy ConvTranspose output differs completely from CPU output with same input

[grybouilli]

Describe the issue

This issue was encountered while testing inference of a Conv-TasNet model.

Given a model composed of a single ConvTranspose with the following attributes:
dilations: 1 group: 1 output_padding: 0 pads: 0,0 strides: 8 weight shape: 512x1x40

Its execution on GPU will produce noisy/erratic output that is completely different from the output it would produce on CPU, given the same input.

Here are some metrics' values comparing GPU vs CPU output for a same input:

and a visual plot of the model output where clearly, gpu output and cpu output don't match:

The phone used is Samsung S24 equipped with the Snapdragon 8 Gen 2 SM8550.

To reproduce

With the provided minimal model : minimal_convtranspose.tar.gz and the following code which should be executed as ./program [cpu, gpu], one can easily obtain and compare the model's output on gpu and cpu:

#include <onnxruntime_cxx_api.h>

#include <array>
#include <iostream>
#include <numeric>
#include <random>
#include <vector>

template <typename T, size_t Dim>
struct OrtTensorBuffer {
    OrtTensorBuffer(const Ort::MemoryInfo&           mem_info,
                    const std::array<int64_t, Dim>&& tensor_shape) :
        shape{tensor_shape},
        buffer_memory(std::accumulate(tensor_shape.begin(),
                                      tensor_shape.end(),
                                      int64_t{1},
                                      std::multiplies<int64_t>{}),
                      0.f),
        tensor{} {
        if constexpr (std::same_as<T, float>) {
            tensor =
                Ort::Value::CreateTensor(mem_info,
                                         buffer_memory.data(),
                                         buffer_memory.size() * sizeof(float),
                                         shape.data(),
                                         shape.size(),
                                         ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
        } else if constexpr (std::same_as<T, uint16_t>) {
            tensor = Ort::Value::CreateTensor(
                mem_info,
                buffer_memory.data(),
                buffer_memory.size() * sizeof(uint16_t),
                shape.data(),
                shape.size(),
                ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16);
        }
    }

    std::array<int64_t, Dim> shape;
    std::vector<T>           buffer_memory;
    Ort::Value               tensor;
};

constexpr size_t        upto           = 20;
static constexpr size_t in_shape_size  = 2;
static constexpr size_t out_shape_size = 3;
#define IN_SHAPE  {2, 512, 2002}
#define OUT_SHAPE {2, 1, 16048}

int main(int argc, char** argv) {
    Ort::Env            env(ORT_LOGGING_LEVEL_VERBOSE, "convtranspose_debug\n");
    Ort::SessionOptions session_options;
    std::unordered_map<std::string, std::string> qnn_options;
    qnn_options["backend_type"] =  argv[1];
    session_options.AppendExecutionProvider("QNN", qnn_options);
    session_options.SetIntraOpNumThreads(1);
    session_options.SetGraphOptimizationLevel(
        GraphOptimizationLevel::ORT_ENABLE_ALL);
    session_options.SetLogSeverityLevel(0);
    session_options.SetLogId("ort_session");
    Ort::Session session =
        Ort::Session(env, "convtranspose.onnx", session_options);

    Ort::MemoryInfo memory_info =
        Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
    OrtTensorBuffer<float, 3> input_dummy{memory_info, IN_SHAPE};
    OrtTensorBuffer<float, 3> output{memory_info, OUT_SHAPE};

    // input is a sequence of random values

    std::mt19937                          rng(42);
    std::uniform_real_distribution<float> dist(-1.0f, 1.0f);
    std::generate(input_dummy.buffer_memory.begin(),
                  input_dummy.buffer_memory.end(),
                  [&] { return dist(rng); });

    Ort::AllocatorWithDefaultOptions allocator;
    auto input_name_alloc      = session.GetInputNameAllocated(0, allocator);
    auto output_name_alloc     = session.GetOutputNameAllocated(0, allocator);
    const char* input_names[]  = {input_name_alloc.get()};
    const char* output_names[] = {output_name_alloc.get()};

    Ort::IoBinding bindings{session};
    bindings.ClearBoundInputs();
    bindings.BindInput(input_names[0], input_dummy.tensor);
    bindings.ClearBoundOutputs();
    bindings.BindOutput(output_names[0], output.tensor);
    session.Run(Ort::RunOptions{nullptr}, bindings);
    for (const auto e : output.buffer_memory) {
        std::cout << e << ",";
    }
    std::cout << std::endl << "Job done" << std::endl;

    return 0;
}

Urgency

Pretty urgent, this is part of a research project.

Platform

Android

OS Version

16

ONNX Runtime Installation

Built from Source

Compiler Version (if 'Built from Source')

Android clang version 21.0.0

Package Name (if 'Released Package')

None

ONNX Runtime Version or Commit ID

3f93e1f

ONNX Runtime API

C++/C

Architecture

X64

Execution Provider

Other / Unknown

Execution Provider Library Version

QNN EP

[grybouilli]

Adrian Lizarraga (@adrianlizarraga) Could you maybe have a look at this please ?