Error between runner-versions on pre-configured tart images.

[mschmieder]

TART_EXECUTOR_INSTALL_GITLAB_RUNNER is useful for ensuring a Tart VM has gitlab-runner available, but it appears to skip installation when any gitlab-runner binary is already present in the image.

That leaves a version mismatch unresolved when the Tart executor host runs a newer GitLab Runner than the guest image. In our case, newer host-side GitLab Runner versions generate internal helper commands that the older guest-side gitlab-runner does not understand.

The requested behavior is not necessarily a "force install" flag. A better fit would be version-aware reconciliation:

• If TART_EXECUTOR_INSTALL_GITLAB_RUNNER is set to a concrete version, install or update the guest binary when the existing version does not match.
• If it is set to latest, install or update when the guest binary is not the latest available version.
• If it is set to brew, run the Homebrew install or upgrade path rather than only checking that a binary exists.
• If it is set to curl, ensure the curl-installed binary is current according to the selected behavior.
• Skip only when the existing guest-side runner already satisfies the requested state.

Environment

• gitlab-tart-executor: observed with 1.28.0
• Host runner: GitLab Runner 18.10.0 or newer
• Guest image: Tart macOS image with an older preinstalled gitlab-runner
• Executor configuration uses TART_EXECUTOR_INSTALL_GITLAB_RUNNER=<version>

Problem

When the Tart image already contains gitlab-runner, the executor appears to skip installation even if TART_EXECUTOR_INSTALL_GITLAB_RUNNER requests a specific version.

This means the host and guest runner binaries can remain incompatible. The host runner generates the job scripts and internal commands, while the guest-side runner binary executes commands such as gitlab-runner artifacts-uploader.

Starting with GitLab Runner 18.10.0, the generated artifact upload command can include --timeout. Older guest-side runner versions do not support that option.

Error

Incorrect Usage: flag provided but not defined: -timeout
NAME:
   gitlab-runner artifacts-uploader - create and upload build artifacts (internal)
USAGE:
   gitlab-runner artifacts-uploader [command options] [arguments...]
OPTIONS:
FATAL: flag provided but not defined: -timeout
   --id value                        The build ID to download and upload artifacts for (default: "14914430309") [$CI_JOB_ID]
   --token value                     Build token (default: "[MASKED]") [$CI_JOB_TOKEN]
   --url value                       GitLab CI URL (default: "https://gitlab.com/") [$CI_SERVER_URL]
   --tls-ca-file value               File containing the certificates to verify the peer when using HTTPS (default: "/var/tmp/builds/strykercorp/robotics/geometry-modeling/meshlib.tmp/CI_SERVER_TLS_CA_FILE") [$CI_SERVER_TLS_CA_FILE]
   --tls-cert-file value             File containing certificate for TLS client auth with runner when using HTTPS [$CI_SERVER_TLS_CERT_FILE]
   --tls-key-file value              File containing private key for TLS client auth with runner when using HTTPS [$CI_SERVER_TLS_KEY_FILE]
   --path value                      Add paths to archive (default: "[${OSEP_CPP_INSTALL_BASE_DIR}, ${OSEP_CPP_TEST_OUTPUT_BASE_DIR}]")
   --exclude value                   Exclude paths from the archive
   --untracked                       Add git untracked files
   --verbose                         Detailed information
   --transfer-meter-frequency value  If set to more than 0s it enables an interactive transfer meter (default: "0s") [$TRANSFER_METER_FREQUENCY]
   --generate-artifacts-metadata
   --runner-id value                 (default: "0")
   --repo-url value
   --repo-digest value
   --job-name value
   --executor-name value
   --runner-name value
   --metadata-parameter value
   --started-at value
   --ended-at value
   --schema-version value
   --name value                      The name of the archive (default: "artifacts")
   --expire-in value                 When to expire artifacts (default: "1 day")
   --artifact-format value           Format of generated artifacts (default: "zip")
   --artifact-type value             Type of generated artifacts (default: "archive")
   --compression-level value         Compression level (fastest, fast, default, slow, slowest) [$ARTIFACT_COMPRESSION_LEVEL]

Process exited with status 1

Observed Behavior

1. The Tart guest image already contains gitlab-runner.
2. The executor starts the job.
3. TART_EXECUTOR_INSTALL_GITLAB_RUNNER=<version> is set.
4. The executor detects that gitlab-runner exists and skips installation.
5. The older guest runner executes host-generated helper commands.
6. Artifact upload fails because the older guest runner does not support --timeout.

Expected Behavior

When TART_EXECUTOR_INSTALL_GITLAB_RUNNER is configured, the executor should verify whether the existing guest runner satisfies the requested install mode.

For a concrete version:

TART_EXECUTOR_INSTALL_GITLAB_RUNNER=18.9.0

The executor should run something equivalent to:

installed_version="$(gitlab-runner --version | awk '/Version:/ {print $2}')"
test "$installed_version" = "18.9.0" || install_gitlab_runner_18_9_0

For latest, the executor should compare the installed version to the latest available runner version and update if needed.

For brew, the executor should use Homebrew install or upgrade semantics.

The operation should remain idempotent: if the requested version is already installed, nothing should change.

Why This Matters

The host runner and guest runner binaries are part of the same job execution path. A guest image can easily contain an older preinstalled runner, especially when Tart images are reused over time.

In this specific case, GitLab Runner 18.10.0 and newer can emit artifacts-uploader --timeout, while older guest-side runner versions fail with:

flag provided but not defined: -timeout

Pinning the host runner to 18.9.0 works around the issue because 18.9.0 is the newest version we found before the artifact uploader --timeout option was introduced. That is only a workaround; the long-term fix is to keep the guest runner compatible with the host runner.

Workaround

Current workaround options are:

• Pin the host GitLab Runner to 18.9.0.
• Build Tart images without a preinstalled gitlab-runner, allowing TART_EXECUTOR_INSTALL_GITLAB_RUNNER to install it.
• Manually update the preinstalled guest runner in every Tart image.

The first option prevents use of newer GitLab Runner releases. The other two options move version reconciliation outside the executor, which makes image maintenance more fragile.

Request

Please consider changing TART_EXECUTOR_INSTALL_GITLAB_RUNNER from an existence check to a version-aware reconciliation step.

The important behavior is:

• major.minor.patch: install that exact version when missing or mismatched.
• latest: install or update to the latest available version.
• brew: install or upgrade through Homebrew.
• curl: install or update through the curl-based installer.
• Existing matching versions should be left unchanged.

This would make Tart runner images safer to reuse and would avoid subtle host/guest GitLab Runner incompatibilities during artifact upload and other internal runner operations.

[mschmieder]

I just looked at the current open MRs and I think #142 might be already part of the solution.