sbt test doesn't find the .so

[holdenk]

Am I missing something to make the dependency work ( high-performance-spark/high-performance-spark-examples#108 ) ?

[pomadchin]

👋 @holdenk

Hmm, the project configuration can be cleaned up (.dependsOn(native % Runtime) is enough); but in general I think it looks good.

I'm wondering, does the lib name match to the actually compiled .so file? oh wow, it looks like it is just tests.

I can take a more closer look a bit later today.

[holdenk]

Awesome thanks!

On core/test I get:

           ^
[info] loading settings for project root from build.sbt ...
[info] set current project to examples (in build file:/home/holden/repos/high-performance-spark-examples/)
[info] sbt server started at local:///home/holden/.sbt/1.0/server/28731a8007ab2c35e086/sock
[info] started sbt server
sbt:examples> core/testOnly com.highperformancespark.examples.ffi.NativeExampleSuite
[info] Building library with native build tool CMake
[info] Using CMake version 3.23.1
[error] CMake Warning:
[error]   Ignoring extra path from command line:
[error]    "/home/holden/repos/high-performance-spark-examples/native/target/native/x86_64-linux/build/323"
[info] -- JNI include directories: /usr/lib/jvm/java-11-openjdk-amd64/include;/usr/lib/jvm/java-11-openjdk-amd64/include/linux;/usr/lib/jvm/java-11-openjdk-amd64/include
[info] -- Configuring done
[info] -- Generating done
[info] -- Build files have been written to: /home/holden/repos/high-performance-spark-examples/native/target/native/x86_64-linux/build
[info] Consolidate compiler generated dependencies of target libhigh-performance-spark0
[info] [ 50%] Building C object CMakeFiles/libhigh-performance-spark0.dir/c/sum.c.o
[info] [ 50%] Building C object CMakeFiles/libhigh-performance-spark0.dir/c/sum_wrapper.c.o
[info] [ 75%] Building C object CMakeFiles/libhigh-performance-spark0.dir/c/sumf_wrapper.c.o
[info] [100%] Linking C shared library liblibhigh-performance-spark0.so
[info] [100%] Built target libhigh-performance-spark0
[info] -- Install configuration: "Release"
[info] -- Installing: /home/holden/repos/high-performance-spark-examples/native/target/native/x86_64-linux/bin/./liblibhigh-performance-spark0.so
[success] Library built in /home/holden/repos/high-performance-spark-examples/native/target/native/x86_64-linux/bin/liblibhigh-performance-spark0.so

Then the test fails with:

23/03/29 10:49:37 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 40283, None)
[info] com.highperformancespark.examples.ffi.NativeExampleSuite *** ABORTED ***
[info]   java.lang.UnsatisfiedLinkError: Native library lib high-performance-spark0.so (/native/x86_64-linux/libhigh-performance-spark0.so) cannot be found on the classpath.
[info]   at com.highperformancespark.examples.ffi.SumJNI$.loadPackaged$1(SumJNI.scala:6)
[info]   at com.highperformancespark.examples.ffi.SumJNI$.load$1(SumJNI.scala:6)
[info]   at com.highperformancespark.examples.ffi.SumJNI$.<init>(SumJNI.scala:6)
[info]   at com.highperformancespark.examples.ffi.SumJNI$.<clinit>(SumJNI.scala)
[info]   at com.highperformancespark.examples.ffi.SumJNI.<init>(SumJNI.scala:6)
[info]   at com.highperformancespark.examples.ffi.NativeExampleSuite.$anonfun$new$1(NativeExample.scala:19)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   ...
[error] Uncaught exception when running com.highperformancespark.examples.ffi.NativeExampleSuite: java.lang.UnsatisfiedLinkError: Native library libhigh-performance-spark0.so (/native/x86_64-linux/libhigh-performance-spark0.so) cannot be found on the classpath.
[error] stack trace is suppressed; run last core / Test / testOnly for the full output
23/03/29 10:49:37 INFO SparkContext: Invoking stop() from shutdown hook
23/03/29 10:49:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

[holdenk]

I've tried a few different variants with the nativeloader decorator since it looks like it's looking for "libhigh..." but CMake is making "liblibhigh..." if I add one "lib" it seems to still just look for "libhigh" and if I add "liblibhigh" it seems to look for "libliblibhigh".

[pomadchin]

I'll close it as a resolved question via holdenk/high-performance-spark-examples#1

But don't hesitate to reopen it / open new issues!