Implement event watcher TLS and auth token generation form custom binary #92

sudiptob2 · 2026-01-15T20:22:35Z

EnricoMi · 2026-01-19T16:00:24Z

scripts/submitArmadaSpark.sh

-    AUTH_ARG=" --conf spark.armada.auth.token=$ARMADA_AUTH_TOKEN"
-else
-    AUTH_ARG=""
+    AUTH_ARGS+=("--conf" "spark.armada.auth.token=$ARMADA_AUTH_TOKEN")


Is it safe to have the token on the command args? Should we document that this is discouraged and signin binaries should be used for secure setup?

Improved documentation to warn users

EnricoMi · 2026-01-19T16:05:28Z

scripts/submitArmadaSpark.sh

+    --conf spark.armada.container.image=$IMAGE_NAME
+    --conf spark.armada.queue=$ARMADA_QUEUE
+    --conf spark.armada.lookouturl=${ARMADA_LOOKOUT_URL:-http://localhost:30000}
+    --conf spark.kubernetes.file.upload.path=/tmp


Do we want /tmp to be configurable?

EnricoMi · 2026-01-19T16:06:05Z

scripts/submitArmadaSpark.sh

+    --deploy-mode $DEPLOY_MODE
+    --name $NAME
+    $CLASS_PROMPT $CLASS_ARG
+    --conf spark.home=/opt/spark


/opt/spark should be configurable, same below for /opt/spark/bin/spark-submit. Can be done in a separate PR, though. SPARK_HOME should be used to find Spark binaris, which might fallback to /opt/spark like ${SPARK_HOME:-/opt/spark.

Will implement in a separate PR, tracking in G-Research/spark#165

EnricoMi · 2026-01-19T16:10:39Z

src/main/scala/org/apache/spark/deploy/armada/Config.scala

+  val ARMADA_AUTH_SIGNIN_ARGS: OptionalConfigEntry[String] =
+    ConfigBuilder("spark.armada.auth.signin.args")
+      .doc(
+        "Space-separated arguments to pass to the signin binary when obtaining an authentication token."


Can arguments be quoted? Do we need escaping?

Updated the parsing logic and docs

GeorgeJahad · 2026-01-20T17:51:00Z

This approach seems a little to specific to the signin binary and our internal resources. armada-spark is an open source project after all.

What if we just searched for an optional script called "/opt/spark/extraFiles/getAuth.sh"? If it exists we run it and get the token. If not, we assume no auth is required.

sudiptob2 · 2026-01-20T18:11:44Z

This approach seems a little to specific to the signin binary and our internal resources. armada-spark is an open source project after all.

What if we just searched for an optional script called "/opt/spark/extraFiles/getAuth.sh"? If it exists we run it and get the token. If not, we assume no auth is required.

@GeorgeJahad Sounds reasonable to me. After all, not everyone will have a similar binary and might need extra steps to obtain the token.

GeorgeJahad · 2026-01-20T18:39:08Z

Sounds reasonable to me.

Ok then please move forward with that. Also, @EnricoMi made a good point about: "spark.armada.auth.token=$ARMADA_AUTH_TOKEN"

What do you think of removing "spark.armada.auth.token" entirely? If the getAuth.sh script exists, it creates the token when needed.

sudiptob2 · 2026-01-20T18:41:32Z

What do you think of removing "spark.armada.auth.token" entirely? If the getAuth.sh script exists, it creates the token when needed.

Agree, and I am just implementing that

Signed-off-by: Sudipto Baral <[email protected]> # Conflicts: # src/main/scala/org/apache/spark/deploy/armada/Config.scala

Signed-off-by: Sudipto Baral <[email protected]>

…d of configurable options Signed-off-by: Sudipto Baral <[email protected]>

EnricoMi · 2026-01-21T08:44:34Z

I'd prefer a config option over a script in a magic place. All logic should be configurable in one place, not a mixture of config and filesystem.

But I agree, it should be a single option pointing to a script without any arguments.

Signed-off-by: Sudipto Baral <[email protected]>

GeorgeJahad · 2026-01-21T22:45:35Z

docs/architecture.md

 They can be set in the [conf](../conf/spark-defaults.conf) file.

- `spark.armada.auth.token` - Armada auth token, (specific to the OIDC server being used.)
+- `ARMADA_AUTH_TOKEN` - Environment variable for Armada auth token (for initial submission). For further authentication between driver and executors, configure `spark.armada.auth.script.path` to point to an authentication script.


I think it is more precise to say:

Environment variable for Armada auth token (for cases when authentication script is unavailable). Otherwise configure `spark.armada.auth.script.path` to point to an authentication script.

GeorgeJahad · 2026-01-21T22:49:40Z

scripts/config.sh.example

@@ -0,0 +1,47 @@
+#!/bin/bash


I don't think I want to include this script in this PR. We will eventually have to do a better job of documenting things, but I don't want to do it here.

GeorgeJahad · 2026-01-21T22:58:54Z

src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaUtils.scala

+        conf.flatMap(_.get(Config.ARMADA_AUTH_SCRIPT_PATH)) match {
+          case Some(scriptPath) =>
+            val authScript = new java.io.File(scriptPath)
+            if (authScript.exists() && authScript.canExecute) {


I think we should throw an error if the script is not executable.

GeorgeJahad · 2026-01-21T23:00:16Z

src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaUtils.scala

+            val authScript = new java.io.File(scriptPath)
+            if (authScript.exists() && authScript.canExecute) {
+              try {
+                val token = Seq("sh", authScript.getAbsolutePath).!!.trim


since the script is executable, do you need the "sh" arg?

GeorgeJahad · 2026-01-21T23:00:46Z

src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaUtils.scala

+            if (authScript.exists() && authScript.canExecute) {
+              try {
+                val token = Seq("sh", authScript.getAbsolutePath).!!.trim
+                if (token.nonEmpty) Some(token) else None


I would throw an error if the token is empty

GeorgeJahad · 2026-01-21T23:05:09Z

src/test/scala/org/apache/spark/scheduler/cluster/armada/ArmadaClusterManagerBackendSuite.scala

+  }
+
+  test("getAuthToken returns None when script path config not set") {
+    val token = org.apache.spark.deploy.armada.submit.ArmadaUtils.getAuthToken(Some(sparkConf))


maybe first that the config is not set, just to be sure.

sudiptob2 · 2026-01-22T15:46:29Z

@GeorgeJahad In this implementation ARMADA_AUTH_TOKEN and the auth script are not complementary:

ARMADA_AUTH_TOKEN: Used by the wrapper Docker container for initial submission to the Armada server (submission-time auth).
Auth script (ARMADA_AUTH_SCRIPT_PATH): Used by driver pods for authentication from inside the cluster (runtime auth).

I did not pass ARMADA_AUTH_TOKEN to driver pods by design because

Scenario 1: If we pass it, the driver pod would always see ARMADA_AUTH_TOKEN set and use it, never falling back to the auth script. This breaks the script-based flow.

Scenario 2: If we don't set ARMADA_AUTH_TOKEN in the wrapper pod and try to rely only on the script, the initial submission fails since the wrapper needs it to submit the job.

So I chose this design so that ARMADA_AUTH_TOKEN stays in the wrapper pod only, and driver pods use the auth script. Let's say anyone is not using our Submit script and they are submitting locally without Docker, then they wouldn't need to pass ARMADA_AUTH_TOKEN and they can just rely on the auth script, as there wouldn't be any Docker wrapper.

Any thoughts on a different approach?

GeorgeJahad · 2026-01-22T16:57:04Z

Any thoughts on a different approach?

So the problem is that our internal environment is a bit weird because the signin exec doesn't work when the docker image is running on the devpod, (so we have to use the env var there,) but we don't want to pass that env var to the driver, (because that exposes it to the lookout ui.) So we want the driver to use in the signin executable.

And we don't want to have to explain that wierdness to our non-internal users. So how about this: Let's remove all reference to the ARMADA_AUTH_TOKEN from the scala code, and move it into the script.

So our internal script will check the env var first, and then call signin if it doesn't exist.

But from the perspective of the scala code, everyone has to use the "spark.armada.auth.script.path" to get the token.

What do you think?

GeorgeJahad · 2026-01-23T00:29:04Z

src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaUtils.scala

+                val token = Seq("sh", authScript.getAbsolutePath).!!.trim
+                if (token.nonEmpty) Some(token) else None
+              } catch {
+                case e: Exception =>


I would throw an error if there is an exception, or if the script returns a non 0 exit code

Signed-off-by: Sudipto Baral <[email protected]>

sudiptob2 · 2026-01-23T18:21:19Z

@GeorgeJahad Thanks for the review. I updated the PR addressing the comments. Let me know how it looks.

GeorgeJahad · 2026-01-23T22:42:39Z

scripts/init.sh

    echo '   PYTHON_SCRIPT=/opt/spark/examples/src/main/python/pi.py'
    echo '   SCALA_CLASS=org.apache.spark.examples.SparkPi'
    echo '   CLASS_PATH=local:///opt/spark/extraFiles/spark-examples_2.12-3.5.3.jar'
+    echo '   # Auth: Set ARMADA_AUTH_SCRIPT_PATH for authentication'


Nit: I would remove this line, cause it can't really be set in config.sh, can it?

We can configure it in config.sh,

armada-spark/scripts/submitArmadaSpark.sh

Lines 125 to 128 in a035a90

# Add auth script path if configured

if [ "$ARMADA_AUTH_SCRIPT_PATH" != "" ]; then

SPARK_SUBMIT_ARGS+=("--conf" "spark.armada.auth.script.path=$ARMADA_AUTH_SCRIPT_PATH")

fi

nice, ok, thanks!

GeorgeJahad

looks good, (i found one last nit in init.sh)

sudiptob2 marked this pull request as ready for review January 15, 2026 23:14

sudiptob2 requested a review from GeorgeJahad January 15, 2026 23:14

EnricoMi reviewed Jan 19, 2026

View reviewed changes

sudiptob2 force-pushed the armada-token-using-signin branch from 795f577 to 3e698f3 Compare January 20, 2026 15:36

sudiptob2 requested a review from EnricoMi January 20, 2026 15:47

sudiptob2 added 6 commits January 20, 2026 14:03

add event watcher TLS and auth token generation form custom binary

7e87dd9

Signed-off-by: Sudipto Baral <[email protected]> # Conflicts: # src/main/scala/org/apache/spark/deploy/armada/Config.scala

add some unit test

6f99d6c

Signed-off-by: Sudipto Baral <[email protected]>

configurable block manager port

f865a18

Signed-off-by: Sudipto Baral <[email protected]>

submit script imporvements

437640c

Signed-off-by: Sudipto Baral <[email protected]>

improve arg parsing

06aae03

Signed-off-by: Sudipto Baral <[email protected]>

Add security warnings for ARMADA_AUTH_TOKEN usage

a008803

Signed-off-by: Sudipto Baral <[email protected]>

sudiptob2 force-pushed the armada-token-using-signin branch 2 times, most recently from a2b85e1 to f13e088 Compare January 20, 2026 19:08

sudiptob2 marked this pull request as draft January 20, 2026 21:29

Simplify authentication to use standard getAuthToken.sh script instea…

919c378

…d of configurable options Signed-off-by: Sudipto Baral <[email protected]>

sudiptob2 force-pushed the armada-token-using-signin branch from f13e088 to 919c378 Compare January 20, 2026 22:18

sudiptob2 marked this pull request as ready for review January 20, 2026 22:34

make authscript path configurable

d86e156

Signed-off-by: Sudipto Baral <[email protected]>

GeorgeJahad reviewed Jan 21, 2026

View reviewed changes

GeorgeJahad reviewed Jan 23, 2026

View reviewed changes

sudiptob2 added 2 commits January 23, 2026 10:42

remove ARMADA_AUTH_TOKEN checking, will be handled by the script itself

c361659

Signed-off-by: Sudipto Baral <[email protected]>

improve getAuthToken and add unit tests

6bf4ec1

Signed-off-by: Sudipto Baral <[email protected]>

sudiptob2 marked this pull request as draft January 23, 2026 16:31

sudiptob2 added 4 commits January 23, 2026 12:14

unit test executeScript method

4231941

Signed-off-by: Sudipto Baral <[email protected]>

remove config.sh example

5de4df7

Signed-off-by: Sudipto Baral <[email protected]>

revert test changes as no longer using this file

53dd7b8

Signed-off-by: Sudipto Baral <[email protected]>

remove auth token form the scripts help

a035a90

Signed-off-by: Sudipto Baral <[email protected]>

sudiptob2 marked this pull request as ready for review January 23, 2026 18:20

sudiptob2 requested a review from GeorgeJahad January 23, 2026 18:20

GeorgeJahad reviewed Jan 23, 2026

View reviewed changes

GeorgeJahad approved these changes Jan 23, 2026

View reviewed changes

sudiptob2 merged commit ba16e94 into master Jan 26, 2026
21 of 23 checks passed

sudiptob2 deleted the armada-token-using-signin branch January 26, 2026 05:28

	# Add auth script path if configured
	if [ "$ARMADA_AUTH_SCRIPT_PATH" != "" ]; then
	SPARK_SUBMIT_ARGS+=("--conf" "spark.armada.auth.script.path=$ARMADA_AUTH_SCRIPT_PATH")
	fi

Implement event watcher TLS and auth token generation form custom binary #92

Implement event watcher TLS and auth token generation form custom binary #92

Uh oh!

Conversation

sudiptob2 commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad commented Jan 20, 2026

Uh oh!

sudiptob2 commented Jan 20, 2026

Uh oh!

GeorgeJahad commented Jan 20, 2026

Uh oh!

sudiptob2 commented Jan 20, 2026

Uh oh!

EnricoMi commented Jan 21, 2026

Uh oh!

GeorgeJahad Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sudiptob2 commented Jan 22, 2026

Uh oh!

GeorgeJahad commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sudiptob2 commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sudiptob2 commented Jan 15, 2026 •

edited

Loading

GeorgeJahad Jan 21, 2026 •

edited

Loading

GeorgeJahad Jan 21, 2026 •

edited

Loading