Generate the list of boards to build dynamically

[pietroalbini]

Before this PR, we used the build-boards.yml reusable workflow to list which boards were supposed to be built, and then invoked the build-one.yml reusable workflow for each of them. Apart from the extra indirection of two reusable workflows, this meant every time a new app is added we had to remember add it to the workflow.

This PR fixes that by merging the two reusable workflows into one, and generating the build matrix dynamically. There is now a job that runs before everything else, which clones the repo and runs the build/gha-build-boards-matrix.py script. The script scans the repository for app.tomls and generates the build matrix in JSON format. The build jobs then depend on the scheduler job, and set their build matrix to that job's output.

To support building a subset of boards (during releases) the script optionally accepts a list of paths, and it will look for app.tomls only in them (for example build/gha-build-boards-matrix.py app/sidecar app/gimlet). You can play around with the script locally, it doesn't require any credentials nor dependencies (except for git and a recent Python).

I validated that all jobs currently executed are still executed in the new setup, and while checking so discovered that we were instead missing jobs in the current setup (ignore the , -> \t changes that sometime show up, since that field was now automatically generated I opted to remove a tr invocation in the workflow by properly separating the fields):

git show master:.github/workflows/build-boards.yml | yq '.jobs[].strategy.matrix.include' | jq -s 'flatten | sort | del(.[].build)' > old.json
build/gha-build-boards-matrix.py | jq -s 'flatten | sort | del(.[].job_name)' > new.json
diff -u old.json new.json

Output of diff

--- old.json	2025-10-02 12:29:22.262667296 +0200
+++ new.json	2025-10-02 12:29:02.788403166 +0200
@@ -25,11 +25,21 @@
     "image": "default"
   },
   {
+    "app_name": "demo-stm32g031-nucleo",
+    "app_toml": "app/demo-stm32g0-nucleo/app-g031.toml",
+    "image": "default"
+  },
+  {
     "app_name": "demo-stm32g070-nucleo",
     "app_toml": "app/demo-stm32g0-nucleo/app-g070.toml",
     "image": "default"
   },
   {
+    "app_name": "demo-stm32g070-nucleo-mini",
+    "app_toml": "app/demo-stm32g0-nucleo/app-g070-mini.toml",
+    "image": "default"
+  },
+  {
     "app_name": "demo-stm32h743-nucleo",
     "app_toml": "app/demo-stm32h7-nucleo/app-h743.toml",
     "image": "default"
@@ -45,6 +55,11 @@
     "image": "default"
   },
   {
+    "app_name": "donglet-g031-i2c",
+    "app_toml": "app/donglet/app-g031-i2c.toml",
+    "image": "default"
+  },
+  {
     "app_name": "gemini-bu",
     "app_toml": "app/gemini-bu/app.toml",
     "image": "default"
@@ -130,6 +145,26 @@
     "image": "default"
   },
   {
+    "app_name": "gimletlet-ereportlet",
+    "app_toml": "app/gimletlet/app-ereportlet.toml",
+    "image": "default"
+  },
+  {
+    "app_name": "gimletlet-meanwell",
+    "app_toml": "app/gimletlet/app-meanwell.toml",
+    "image": "default"
+  },
+  {
+    "app_name": "gimletlet-mgmt",
+    "app_toml": "app/gimletlet/app-mgmt.toml",
+    "image": "default"
+  },
+  {
+    "app_name": "gimletlet-sidecar-emulator",
+    "app_toml": "app/gimletlet/app-sidecar-emulator.toml",
+    "image": "default"
+  },
+  {
     "app_name": "grapefruit-ruby",
     "app_toml": "app/grapefruit/app-ruby.toml",
     "image": "default"
@@ -142,7 +177,12 @@
   {
     "app_name": "lpc55xpresso",
     "app_toml": "app/lpc55xpresso/app.toml",
-    "image": "a, b"
+    "image": "a\tb"
+  },
+  {
+    "app_name": "medusa-a",
+    "app_toml": "app/medusa/model-a.toml",
+    "image": "default"
   },
   {
     "app_name": "minibar",
@@ -150,14 +190,24 @@
     "image": "default"
   },
   {
+    "app_name": "minibar-net",
+    "app_toml": "app/minibar/app-net.toml",
+    "image": "default"
+  },
+  {
+    "app_name": "oxcon2023g0",
+    "app_toml": "app/oxcon2023g0/app.toml",
+    "image": "default"
+  },
+  {
     "app_name": "oxide-rot-1",
     "app_toml": "app/oxide-rot-1/app.toml",
-    "image": "a, b"
+    "image": "a\tb"
   },
   {
     "app_name": "oxide-rot-1-selfsigned",
     "app_toml": "app/oxide-rot-1/app-dev.toml",
-    "image": "a, b"
+    "image": "a\tb"
   },
   {
     "app_name": "psc-b",
@@ -182,7 +232,7 @@
   {
     "app_name": "rot-carrier",
     "app_toml": "app/rot-carrier/app.toml",
-    "image": "a, b"
+    "image": "a\tb"
   },
   {
     "app_name": "sidecar-b",
@@ -242,7 +292,7 @@
   {
     "app_name": "tests-lpc55xpresso",
     "app_toml": "test/tests-lpc55xpresso/app.toml",
-    "image": "a, b"
+    "image": "a\tb"
   },
   {
     "app_name": "tests-psc",
@@ -252,7 +302,7 @@
   {
     "app_name": "tests-rot-carrier",
     "app_toml": "test/tests-rot-carrier/app.toml",
-    "image": "a, b"
+    "image": "a\tb"
   },
   {
     "app_name": "tests-stm32fx",

The reason why I opted to write the matrix generator in a scripting language rather than Rust (as part of xtask) is the speed of the scheduler job. With the current setup, the job in total takes 4 seconds to run, while running xtask with a full cache would take around a minute. My hunch is that it's better to deal with a scripting language rather than making every CI run a minute slower. I chose Python as the scripting language mostly because it has a TOML parser in its standard library, and thus doesn't require any dependency.

This PR should be reviewed commit-by-commit (the first two commits just move files around, helping git notice renames).

[labbott]

First pass LGTM! I'll let others get a chance to review too.