Tests for Spark

[bkmgit]

Maybe helpful to add tests for Spark using R, Java and Scala interfaces.

[boegel]

@bkmgit Any suggestions for tests to use?

[bkmgit]

The default setup seems to install Spark in standalone mode. Spark has a job submission script format, however if using more than one node on a cluster with a job scheduler, then one first needs to obtain the resources from the job scheduler and then give information about the provisioned nodes to Spark.

Sparkhpc makes it easy to use Pyspark, in particular integrating an HPC scheduler.
Example scripts can be found in the spark on hpc repository.

Some tutorial notes on using installing Spark with Easybuild and then using that installation on a single node.

Spark does include a number of tests using sbt, maven and Python. Each major Spark release uses one version of Scala, so it may be helpful to install this along with Java 8.