Add jupyter

[sudiptob2]

Closes G-Research/spark#128

[GeorgeJahad]

I rushed you to create this PR, so we could demo it, and it was great for that, but now that the demo is over I think we need to make it more user-friendly. I made the following comments with that in mind.

[GeorgeJahad]

in our internal cluster, i had to make the following mods to the default notebook:

   {
    "cell_type": "code",
    "id": "configuration",
    "metadata": {},
    "outputs": [],
    "source": [
     "# Configuration\n",
-    "driver_host = os.environ.get('SPARK_DRIVER_HOST', '10.0.0.80')\n",
-    "driver_port = os.environ.get('SPARK_DRIVER_PORT', '7078')\n",
+    "\n",
+    "auth_token = os.environ.get('ARMADA_AUTH_TOKEN')\n",
+    "driver_host = os.environ.get('SPARK_DRIVER_HOST', '11.2.208.34')\n",
+    "driver_port = os.environ.get('SPARK_DRIVER_PORT', '10060')\n",
     "block_manager_port = os.environ.get('SPARK_BLOCK_MANAGER_PORT', '10061')\n",
-    "armada_master = os.environ.get('ARMADA_MASTER', 'local://armada://host.docker.internal:30002')\n",
-    "armada_queue = os.environ.get('ARMADA_QUEUE', 'test')\n",
-    "image_name = os.environ.get('IMAGE_NAME', 'spark:armada')\n",
+    "armada_master = os.environ.get('ARMADA_MASTER', 'armada://XXX:443')\n",
+    "armada_queue = os.environ.get('ARMADA_QUEUE', 'XXX')\n",
+    "image_name = os.environ.get('IMAGE_NAME', 'XXX/spark:armada')\n",
     "\n",
     "# Find JAR\n",
-    "jar_paths = glob.glob('/opt/spark/jars/armada-cluster-manager_2.13-*-all.jar')\n",
+    "jar_paths = glob.glob('/opt/spark/jars/armada-cluster-manager_2.12-*-all.jar')\n",
     "if not jar_paths:\n",
     "    raise FileNotFoundError(\"Armada Spark JAR not found!\")\n",
     "armada_jar = jar_paths[0]\n",
     "\n",
     "# Generate app ID, required for client mode\n",
-    "app_id = f\"armada-spark-{subprocess.check_output(['openssl', 'rand', '-hex', '3']).decode().strip()}\""
+    "app_id = f\"jupyter-spark-{subprocess.check_output(['openssl', 'rand', '-hex', '3']).decode().strip()}\""
    ]
   },
   {
@@ -97,7 +99,8 @@
   },
    "source": [
     "# Spark Configuration\n",
     "conf = SparkConf()\n",
+    "conf.set(\"spark.armada.auth.token\", auth_token)\n",
     "conf.set(\"spark.master\", armada_master)\n",
     "conf.set(\"spark.submit.deployMode\", \"client\")\n",
     "conf.set(\"spark.app.id\", app_id)\n",
@@ -114,7 +128,6 @@
     "conf.set(\"spark.driver.blockManager.port\", block_manager_port)\n",
     "conf.set(\"spark.home\", \"/opt/spark\")\n",
     "conf.set(\"spark.armada.container.image\", image_name)\n",
-    "conf.set(\"spark.armada.scheduling.nodeUniformity\", \"armada-spark\")\n",
     "conf.set(\"spark.armada.queue\", armada_queue)\n",
     "conf.set(\"spark.kubernetes.file.upload.path\", \"/tmp\")\n",
     "conf.set(\"spark.kubernetes.executor.disableConfigMap\", \"true\")\n",
@@ -127,15 +140,18 @@
     "\n",
     "# Static mode\n",
     "conf.set(\"spark.executor.instances\", \"2\")\n",
-    "conf.set(\"spark.armada.executor.limit.memory\", \"1Gi\")\n",
-    "conf.set(\"spark.armada.executor.request.memory\", \"1Gi\")"
+    "conf.set(\"spark.armada.driver.limit.memory\", \"10Gi\")\n",
+    "conf.set(\"spark.armada.driver.request.memory\", \"10Gi\")\n",
+    "conf.set(\"spark.armada.executor.limit.memory\", \"60Gi\")\n",
+    "conf.set(\"spark.armada.executor.request.memory\", \"60Gi\")\n",
+    "#print (conf.get(\"spark.armada.auth.token\"))"
    ]
   },

[GeorgeJahad]

env var?

[sudiptob2]

The host port is already configurable using JUPYTER_PORT env variable. This is the internal Jupyter port, don't think we need to make this configurable.

armada-spark/scripts/runJupyter.sh

Lines 54 to 56 in 779b8d6

	docker run -d \
	--name armada-jupyter \
	-p ${JUPYTER_PORT}:8888 \

[GeorgeJahad]

The docker file recieves the include_python arg. Probably we shouldn't include jupyter if python is not included.

[GeorgeJahad]

where does that version string come from? Can we just do

 py4j-*src.zip

?

[GeorgeJahad]

this looks like the notebook will be read only.

what i was thinking would be good is if we have a workspace directory that has no checked in code.

If it doesn't contain the notebook file, this script copies that in.

But it doesn't touch it if it exists already.

Then, that directory gets mounted in docker.

[GeorgeJahad]

After your changes stabilize I'd also like this branch tested against the internal cluster.

[GeorgeJahad]

lgtm

[GeorgeJahad]

thanks @sudiptob2 !