cublas tests fail

[drahnr]

They are not failing all the time, sometimes the first few pass, sometimes they all pass.

[Anton-4]

I've narrowed down this issue a bit. The dot and nrm2 operation seem to be the core of the problem.
When ignoring these and running the tests single-threaded, all other tests always succeed.
When one operation fails the CUDA context is broken and all following cuda tests fail.
Due to parallel testing this resulted in a varying number of tests failing with different runs.

[drahnr]

Awesome! I would love to know what is going wrong there exactly :)

Note that @hobofan suspected that the memory allocated does not suffice some constraints required - I did not verify this yet.

[Anton-4]

Can you assign me here?
The underlying CUDA error code is for unknown error so this won't be easy to figure out.
I'll check if it is caused by a problem with memory allocation.

[drahnr]

@Anton-4 right now it seems I can only assign team members, so I am going to do some magic to make this work and add you to a subteam named @spearow/contributor - as soon as you accept that I can assign issues to you :)

I am aware that this is not easy, I'd recommend to start with the API docs NVIDIA provides, maybe unknown is not exactly correct and only there because not all error codes are differentiated properly.

Memory alignment can be checked with assert_eq!(mem::align_of::<Foo>(), 16);

[Anton-4]

Some updates:

• the issue is definitely memory related and most likely happens in coaster self, not coaster-blas
• running the dot test by itself succeeds. When running another test before dot, the dot test fails.
  so my current guess is some memory is not correctly cleaned up after writing.
• some cuda-memcheck errors:

======== Program hit cudaErrorInvalidDevice (error 10) due to "invalid device ordinal" on CUDA API call to cudaEventQuery. 

========= Program hit cudaErrorInvalidDevice (error 10) due to "invalid device ordinal" on CUDA API call to cudaGetLastError. 

========= Invalid __global__ write of size 4
=========     at 0x000004d0 in void dot_kernel<float, float, float, int=128, int=0, int=0>(cublasDotParams<float, float>)
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x705c40000 is out of bounds

Getting a little bit closer every day :)

[drahnr]

Did you try cuda-memcheck --leak-check full to get the origin of the allocations?

This sounds awfully similar to coaster-nn/#11 which seems to points even more towards coaster ... specifically ./src/frameworks/cuda/api/driver/memory.rs and the ffi in there.

Thanks for taking care of this. Much appreciated!

[Anton-4]

relevant output for cuda-memcheck --leak-check ful:

test cuda_f32::it_computes_correct_asum ... ========= Leaked 32768 bytes at 0x705c40000
=========     Saved host backtrace up to driver entry point at cudaMalloc time
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 (cuMemAlloc_v2 + 0x17f) [0x22076f]
=========
========= Leaked 1024 bytes at 0x705a40600
=========     Saved host backtrace up to driver entry point at cudaMalloc time
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 (cuMemAlloc_v2 + 0x17f) [0x22076f]
=========
========= Leaked 112 bytes at 0x705a40400
=========     Saved host backtrace up to driver entry point at cudaMalloc time
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 (cuMemAlloc_v2 + 0x17f) [0x22076f]

========= LEAK SUMMARY: 33904 bytes leaked in 3 allocations

0x705c40000 points to an assembly instruction (lea) any suggestions on how to find the corresponding rust code?

[Anton-4]

updates:

• I checked if all the memory allocated with
  coaster/src/frameworks/cuda/api/driver/memory.rs is freed and this is the case. The total memory allocated here is 48 bytes which is a big difference
  with the 32 kb leaked.
• I also ran the memcheck with an extra option --report-api-errors all, which revealed another error:

  Program hit CUDA_ERROR_NO_BINARY_FOR_GPU (error 209) due to "no kernel image is    available for execution on the device" on CUDA API call to load binary via CUDA runtime.
  

  I haven't been able to find anything useful with this error yet.
• I also set up debugging with lldb in vscode to try and trace the dot operation as deep as possible.
  I still need to replace some macro's so I can trace deeper.

[drahnr]

• The internal stuff here might be caused by some static allocations by the library itself?
• similar issue essentially lacking the PTX JIT compiler so CUDA_ERROR_NO_BINARY_FOR_GPU Theano/libgpuarray#289
• Which macros are you talking about? The rust or some cudblas header macros?

[Anton-4]

• Indeed I've been thinking the same thing
• will check it out
• rust macros

[drahnr]

Using Fedora ptxjitcompiler is provided by the following packages (from negativio17.org nvidia repo)

dnf provides '/usr/lib64/libnvidia-ptxjitcompiler.so.384.90'                                                                                                23:03:42   130 
Last metadata expiration check: 0:09:27 ago on Sun 29 Oct 2017 10:54:39 PM CET.

nvidia-driver-cuda-libs-2:384.90-1.fc26.x86_64 : Libraries for nvidia-driver-cuda
Repo        : @System
Matched from:
Filename    : /usr/lib64/libnvidia-ptxjitcompiler.so.384.90

nvidia-driver-cuda-libs-2:384.90-1.fc26.x86_64 : Libraries for nvidia-driver-cuda
Repo        : fedora-nvidia
Matched from:
Filename    : /usr/lib64/libnvidia-ptxjitcompiler.so.384.90

So this file is present on my test machine, yet this happens too.

---

Did you figure out a way to map the instruction back to a rust function?

If there are only a few suspects, #[no_mangle] might be enough.

I am currently tied to a GTX 460 but I can test later this week on a GTX 1050 if it makes any difference.

[Anton-4]

From what I've read NO_BINARY_FOR_GPU can have many causes.
I will give #[no_mangle] a try.

[drahnr]

The cublas doc also states that CUDA_FORCE_PTX_JIT=1 uses a fallback path based on PTX, so that might be a cross test too.