CUSTOM_ENV_* requirements breaking custom executor with GitLab Runner

[juliengdt]

Context

I’m having trouble using gitlab-tart-executor as a custom executor with GitLab Runner on macOS (local development machine). After several attempts with different Runner and Tart versions, the blocker clearly appears inside gitlab-tart-executor itself, at the level of the CUSTOM_ENV_* environment handling.

The goal is to use CirrusLabs macOS images (Sequoia/Tahoe) in GitLab CI via gitlab-tart-executor on a macOS Apple Silicon machine (M series).

Observed behavior

1. The executor works correctly for the --version command:

/opt/homebrew/bin/gitlab-tart-executor --version
executor version 1.28.0-6e163f8

2. However, even a simple config fails immediately when CUSTOM_ENV_CI_JOB_ID is not present in the environment, including outside of GitLab Runner:

/opt/homebrew/bin/gitlab-tart-executor config
GitLab environment error: CUSTOM_ENV_CI_JOB_ID is missing

3. In GitLab CI, with the Runner configured in executor = "custom" and calling gitlab-tart-executor directly for config_exec / prepare_exec / run_exec / cleanup_exec, the job always fails with:

Preparing the "custom" executor
Using Custom executor with driver tart 1.28.0-6e163f8...
"prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set
WARNING: Cleanup script failed: exit status 2
ERROR: Job failed: exit status 1

4. In the runner logs, you can see the job being picked up and then going straight to cleanup, without any visible Tart VM creation:

Checking for jobs... received ...
Added job to processing list ...
WARNING: Failed to stop VM: tart command returned non-zero exit code: "the specified VM \"gitlab-<job-id>\" does not exist"
WARNING: Failed to delete VM: VM errored: failed to delete VM gitlab-<job-id>: tart command returned non-zero exit code: "the specified VM \"gitlab-<job-id>\" does not exist"
WARNING: VM errored: failed to delete VM gitlab-<job-id>: tart command returned non-zero exit code: "the specified VM \"gitlab-<job-id>\" does not exist"
WARNING: Cleanup script failed: exit status 2
WARNING: Job failed: exit status 1

In other words, gitlab-tart-executor refuses to run config/prepare as long as CUSTOM_ENV_CI_JOB_ID (and potentially other CUSTOM_ENV_* variables) are not defined. The error message also mentions that no --default-image was set.

GitLab Runner configuration (excerpt)

Here is an example of my current runner configuration:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "MacBook-M1Max-Tart"
  url = "https://gitlab.com"
  id = 52312176
  token = "REDACTED"
  executor = "custom"
  environment = ["TART_EXECUTOR_SOFTNET=true", "TART_EXECUTOR_ALWAYS_PULL=false"]

  [runners.feature_flags]
    FF_RESOLVE_FULL_TLS_CHAIN = false

  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

  [runners.custom]
    config_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    config_args = ["config"]

    prepare_exec = "sudo"
    prepare_args = [
      "/opt/homebrew/bin/gitlab-tart-executor", "prepare",
      "--user", "juliengoudet",
      "--concurrency", "1",
      "--cpu", "auto",
      "--memory", "auto"
    ]

    run_exec = "sudo"
    run_args = [
      "/opt/homebrew/bin/gitlab-tart-executor", "run",
      "--user", "juliengoudet"
    ]

    cleanup_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    cleanup_args = ["cleanup"]

The Tart CLI itself (tart) works fine when used directly to launch CirrusLabs macOS images on this machine.

What I already tried

• GitLab Runner 18.8.0 / 18.9.0: same error in CI
  "prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set.
• Downgraded GitLab Runner to 17.11.3: same behavior, i.e. gitlab-tart-executor config and prepare refuse to start without CUSTOM_ENV_CI_JOB_ID being defined.
• Verified sudoers and permissions:
  sudo /opt/homebrew/bin/gitlab-tart-executor --version works without a password, but config still fails with the same CUSTOM_ENV_* error.
• Manual tests with CUSTOM_ENV_CI_JOB_ID defined in the environment before calling gitlab-tart-executor config: in that case, config starts working, which confirms that the hard requirement on CUSTOM_ENV_* is enforced inside the executor.

Problem / questions

From what I can see:

• gitlab-tart-executor 1.28.0-6e163f8 is designed to be invoked in an environment where GitLab provides the CUSTOM_ENV_* variables (in particular CUSTOM_ENV_CI_JOB_ID), but neither Runner 17.11.3 nor 18.9.0 seem to populate those variables in the way the executor expects.
• The binary also refuses to run even basic commands like config unless these variables are present, including when called directly from the shell (outside of GitLab Runner).

This leads to two possible workarounds on my side:

• Introduce a wrapper script that maps standard CI_* environment variables to CUSTOM_ENV_* (e.g. CUSTOM_ENV_CI_JOB_ID="$CI_JOB_ID"), and point the custom executor to this wrapper instead of the binary directly.
• Try to rely on --default-image as hinted in the error message, but the current documentation does not clearly explain how this is supposed to work in the absence of CUSTOM_ENV_*.

My questions:

1. Is this behavior expected for gitlab-tart-executor 1.28.0 (always requiring CUSTOM_ENV_CI_JOB_ID for config/prepare), even when running from the CLI?
2. What is the exact expected contract between GitLab Runner and gitlab-tart-executor regarding environment variables (CUSTOM_ENV_* vs CI_*), especially for GitLab Runner 18.x?
3. Is there a documented way to use the executor without a wrapper (for example by correctly using --default-image or another flag) when CUSTOM_ENV_* is not present?
4. Is there a recommended gitlab-tart-executor version for GitLab Runner 18.x, or a specific Runner version officially supported for 1.28.0?

I’m happy to provide:

• Exact versions of GitLab Runner, Tart, macOS, and the full runner config,
• Additional logs (--debug on the runner, and full gitlab-tart-executor output for prepare/run),

if that helps reproducing or clarifying the expected behavior.

EDIT : for refs

According to the official GitLab Runner Custom executor documentation, the driver is supposed to receive all CI/CD and predefined environment variables prefixed with ‎CUSTOM_ENV_ rather than the raw ‎CI_* names.

The docs explicitly state that:

“Each executed step has access to specific environment variables that provide information about the running job. All stages have the following environment variables available to them:

– Standard CI/CD environment variables, including predefined variables.

– All environment variables provided by the Custom executor Runner host system.

– All services and their available settings. Exposed in JSON format as ‎CUSTOM_ENV_CI_JOB_SERVICES.

Both CI/CD environment variables and predefined variables are prefixed with ‎CUSTOM_ENV_ to prevent conflicts with system environment variables. For example, ‎CI_BUILDS_DIR is available as ‎CUSTOM_ENV_CI_BUILDS_DIR.” ​⁠

In other words, for the latest GitLab Runner and Custom executor implementation, the expected contract is that the driver reads metadata like ‎CI_JOB_ID, ‎CI_PROJECT_DIR, etc. via their ‎CUSTOM_ENV_* counterparts (for example ‎CUSTOM_ENV_CI_JOB_ID, ‎CUSTOM_ENV_CI_PROJECT_DIR), which are injected by GitLab into the driver’s environment for all stages (‎config_exec, ‎prepare_exec, ‎run_exec, ‎cleanup_exec).