[WIP] Add a CI job to run unit tests on an AIX host - #55602
Conversation
AIX/ppc64 has no Bazel, so cannot use to drive unit tests. Mirror the existing AIX/no-Bazel fallback in tasks/schema/generate.py: on AIX, invoke the prebuilt gotestsum binary directly via ctx.run and wrap the invoke Result into a subprocess.CompletedProcess so the rest of the test loop (which reads res.returncode / res.stdout) is unchanged. Non-AIX behavior is unchanged.
ctx.run goes through a shell, so building the command by f-string
interpolation (f'gotestsum {formatted_cmd}') let shell metacharacters
in the command — e.g. | in a -run TestA|TestB regex from
--test-run-name — be interpreted by the shell instead of passed
literally to go test.
Build the exact argv (the same shlex.split list the bazel path passes
to gotestsum) and re-quote it with shlex.join, so the shell parses it
back to that same argv. Verified on AIX: --test-run-name='TestA|TestB'
runs the matching tests instead of failing with 'TestB: not found'.
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
Patch is incorrect: the job executes PR-controlled code as root on a persistent shared host and does not securely authenticate that host.
| # Setup the host (idempotent) then run the unit tests. | ||
| # setup-host.sh is shared with devs (packaging/aix/), run-tests.sh is the | ||
| # unit-test job's script (packaging/aix/ci/). | ||
| - ssh -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "$AIX_AGENT_SRC/packaging/aix/setup-host.sh" |
There was a problem hiding this comment.
This executes setup-host.sh from the commit under review as root on a long-lived host shared by later jobs. A malicious or compromised PR can install persistent code, alter the checkout/toolchain, and contaminate subsequent trusted runs. Provision from trusted code outside the PR and run tests as an unprivileged user, or use a fresh isolated host per job.
| # Trust the host key | ||
| - ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null |
There was a problem hiding this comment.
ssh-keyscan retrieves the key over the same unauthenticated network connection it is meant to secure, so a MITM can substitute its own host key. Store the expected SiteOX host key/fingerprint in a protected CI variable or Vault and verify against that pinned value.
Gitlab CI Configuration ChangesModified Jobsvariables (configuration) variables:
AGENT_API_KEY_ORG2: agent-api-key-org-2
AGENT_APP_KEY_ORG2: agent-app-key-org-2
AGENT_BINARIES_DIR: bin/agent
AGENT_QA_E2E: agent-qa-e2e
ARTIFACT_DOWNLOAD_ATTEMPTS: 2
ATLASSIAN_WRITE: atlassian-write
BTFHUB_ARCHIVE_BRANCH: main
BUCKET_BRANCH: dev
CACHE_COMPRESSION_LEVEL: slowest
CHANGELOG_COMMIT_SHA: ci.datadog-agent.gitlab_changelog_commit_sha
CI_IMAGE_BTF_GEN: v132333024-4b82e653
CI_IMAGE_BTF_GEN_SUFFIX: ''
CI_IMAGE_DOCKER_ARM64: v132333024-4b82e653
CI_IMAGE_DOCKER_ARM64_SUFFIX: ''
CI_IMAGE_DOCKER_X64: v132333024-4b82e653
CI_IMAGE_DOCKER_X64_SUFFIX: ''
CI_IMAGE_GITLAB_AGENT_DEPLOY: v132333024-4b82e653
CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX: ''
CI_IMAGE_LINUX: v132333024-4b82e653
CI_IMAGE_LINUX_SUFFIX: ''
CI_IMAGE_RPM_ARM64: v132333024-4b82e653
CI_IMAGE_RPM_ARM64_SUFFIX: ''
CI_IMAGE_RPM_ARMHF: v132333024-4b82e653
CI_IMAGE_RPM_ARMHF_SUFFIX: ''
CI_IMAGE_RPM_X64: v132333024-4b82e653
CI_IMAGE_RPM_X64_SUFFIX: ''
CI_IMAGE_WIN_LTSC2022_X64: v130938967-3a9f81bc
CI_IMAGE_WIN_LTSC2022_X64_SUFFIX: ''
CI_IMAGE_WIN_LTSC2025_X64: v130938967-3a9f81bc
CI_IMAGE_WIN_LTSC2025_X64_SUFFIX: ''
CLANG_BUILD_VERSION: v60409452-ee70de70
CLANG_LLVM_VER: 12.0.1
CLUSTER_AGENT_BINARIES_DIR: bin/datadog-cluster-agent
CLUSTER_AGENT_CLOUDFOUNDRY_BINARIES_DIR: bin/datadog-cluster-agent-cloudfoundry
CODECOV: codecov
COMPARE_TO_BRANCH: main
CRC_PULL_SECRET: ci.datadog-agent.crc-pull-secret
CWS_INSTRUMENTATION_BINARIES_DIR: bin/cws-instrumentation
DATADOG_AGENT_EMBEDDED_PATH: /opt/datadog-agent/embedded
DDA_CLIENT_TOKEN: dda-feature-flags-client-token
DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS: ci.datadog-agent.dda-feature-flags-client-token
DDA_FEATURE_FLAGS_CI_VAULT_KEY: token
DDA_FEATURE_FLAGS_CI_VAULT_KEY_MACOS: token
DDA_FEATURE_FLAGS_CI_VAULT_PATH: k8s/gitlab-runner-datadog-agent/datadog-agent/$DDA_CLIENT_TOKEN
DDA_FEATURE_FLAGS_CI_VAULT_PATH_MACOS: aws/arn:aws:iam::486234852809:role/ci-datadog-agent/$DDA_CLIENT_TOKEN
DD_AGENT_TESTING_DIR: $CI_PROJECT_DIR/test/new-e2e/tests
DD_PKG_GITLAB_URL: https://artifact-gateway.us1.ddbuild.io/internal/artifact-gateway/api/v4
DEB_GPG_KEY_ID: c0962c7d
DEB_GPG_KEY_NAME: Datadog, Inc. APT key
DEB_RPM_TESTING_BUCKET_BRANCH: testing
DEB_S3_BUCKET: apt.datad0g.com
DEB_TESTING_S3_BUCKET: apttesting.datad0g.com
DISABLE_GIT_CACHE: true
DOCKER_REGISTRY_RO: dockerhub-readonly
DOCKER_REGISTRY_URL: docker.io
DOGSTATSD_BINARIES_DIR: bin/dogstatsd
DYNAMIC_TESTS_BREAKGLASS: dynamic-tests-breakglass
E2E_AZURE: e2e-azure
E2E_COVERAGE_PIPELINE: false
E2E_GCP: e2e-gcp
EXECUTOR_JOB_SECTION_ATTEMPTS: 2
FF_CLEAN_UP_FAILED_CACHE_EXTRACT: true
FF_KUBERNETES_HONOR_ENTRYPOINT: true
FF_SCRIPT_SECTIONS: 1
FF_TIMESTAMPS: true
FF_USE_FASTZIP: true
FF_USE_WINDOWS_JOB_OBJECT: true
GENERAL_ARTIFACTS_CACHE_BUCKET_URL: https://dd-agent-omnibus.s3.amazonaws.com
GET_SOURCES_ATTEMPTS: 2
GIT_DEPTH: 1
GIT_STRATEGY: fetch
GO_TEST_SKIP_FLAKE: 'true'
GPG_TEST_KEY_ID: crypto/k8s/keys/k8s_gitlab-runner-datadog-agent_datadog-agent_testing_signing-key
INSTALLER_TESTING_S3_BUCKET: installtesting.datad0g.com
INSTALL_SCRIPT_API_KEY_ORG2: install-script-api-key-org-2
KERNEL_MATRIX_TESTING_ARM_AMI_ID: ami-0b5f838a19d37fc61
KERNEL_MATRIX_TESTING_X86_AMI_ID: ami-05b3973acf5422348
KITCHEN_INFRASTRUCTURE_FLAKES_RETRY: 2
MACOS_APPLE_APPLICATION_SIGNING: apple-application-signing
MACOS_APPLE_DEVELOPER_ACCOUNT: apple-developer-account
MACOS_APPLE_INSTALLER_SIGNING: apple-installer-signing
MACOS_KEYCHAIN_PWD: ci-keychain
MACOS_S3_BUCKET: dd-agent-macostesting
OMNIBUS_BASE_DIR: /omnibus
OMNIBUS_PACKAGE_DIR: $CI_PROJECT_DIR/omnibus/pkg/
OMNIBUS_PACKAGE_DIR_SUSE: $CI_PROJECT_DIR/omnibus/suse/pkg
PIPELINE_KEY_ALIAS: alias/ci_datadog-agent_pipeline-key
PROCESS_S3_BUCKET: datad0g-process-agent
PYTHONUNBUFFERED: 1
RESTORE_CACHE_ATTEMPTS: 2
RPM_GPG_KEY_ID: b01082d3
RPM_GPG_KEY_NAME: Datadog, Inc. RPM key
RPM_S3_BUCKET: yum.datad0g.com
RPM_TESTING_S3_BUCKET: yumtesting.datad0g.com
RUN_E2E_TESTS: auto
RUN_KMT_TESTS: auto
RUN_UNIT_TESTS: auto
S3_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME/$CI_PIPELINE_ID
S3_CP_CMD: aws s3 cp $S3_CP_OPTIONS
S3_CP_OPTIONS: --no-progress --region us-east-1 --sse AES256
S3_DD_AGENT_OMNIBUS_BTFS_URI: s3://dd-agent-omnibus/btfs
S3_DD_AGENT_OMNIBUS_JAVA_URI: s3://dd-agent-omnibus/openjdk
S3_DD_AGENT_OMNIBUS_LLVM_URI: s3://dd-agent-omnibus/llvm
S3_DSD6_URI: s3://dsd6-staging
S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
S3_OMNIBUS_GIT_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-git-cache-build-stable
S3_PERMANENT_ARTIFACTS_URI: s3://dd-ci-persistent-artefacts-build-stable/$CI_PROJECT_NAME
S3_PROJECT_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME
S3_RELEASE_ARTIFACTS_URI: s3://dd-release-artifacts/$CI_PROJECT_NAME/$CI_PIPELINE_ID
S3_RELEASE_INSTALLER_ARTIFACTS_URI: s3://dd-release-artifacts/datadog-installer/$CI_PIPELINE_ID
S3_SBOM_STORAGE_URI: s3://sbom-root-us1-ddbuild-io/$CI_PROJECT_NAME/$CI_PIPELINE_ID
SECRET_GENERIC_CONNECTOR_BINARIES_DIR: bin/secret-generic-connector
+ SITEOX_AIX_CI_RUNNER: siteox-aix-ci-runner
SKIP_WINDOWS: 'false'
SLACK_AGENT: slack-agent-ci
SMP_ACCOUNT: smp
STATIC_BINARIES_DIR: bin/static
SYSTEM_PROBE_BINARIES_DIR: bin/system-probe
TEST_KEYS_URL: apttesting.datad0g.com/test-keys
VCPKG_BLOB_SAS_URL: vcpkg-blob-sas
VIRUS_TOTAL: virus-total
WINDOWS_BUILDS_S3_BUCKET: $WIN_S3_BUCKET/builds
WINDOWS_SYMBOLS_S3_BUCKET: pipelines/windows-symbols
WINDOWS_TESTING_S3_BUCKET: pipelines/A7/$CI_PIPELINE_ID
WINGET_PAT: winget-pat
WIN_S3_BUCKET: dd-agent-mstestingAdded Jobs.aix_remote.aix_remote:
before_script:
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- touch ~/.ssh/aix_ci_key && chmod 600 ~/.ssh/aix_ci_key
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SITEOX_AIX_CI_RUNNER key > ~/.ssh/aix_ci_key
|| exit $?
- chmod 600 ~/.ssh/aix_ci_key
- ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null
- 'echo "=== AIX: cloning agent source at $CI_COMMIT_SHA ==="'
- "ssh -T -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST bash -s <<EOF\n\
set -eu\nexport PATH=/opt/freeware/bin:/usr/bin:\\$PATH\nif [ ! -d \"$AIX_AGENT_SRC/.git\"\
\ ]; then\n git clone --progress \"$AIX_GIT_REMOTE\" \"$AIX_AGENT_SRC\"\nfi\n\
cd \"$AIX_AGENT_SRC\"\ngit clean -df\ngit reset --hard\ngit fetch --progress origin\n\
# The commit under test may not be reachable from origin yet (e.g. a\n# brand-new\
\ branch push); fetch the branch ref as a fallback.\ngit fetch --progress origin\
\ \"$CI_COMMIT_REF_NAME\" || true\ngit checkout \"$CI_COMMIT_SHA\"\nEOF\n"
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
needs: []
resource_group: aix_ci_host
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- 'echo "=== AIX: provisioning host (setup-host.sh) ==="'
- ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/setup-host.sh"
- 'echo "=== AIX: running unit tests (run-tests.sh) ==="'
- ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/ci/run-tests.sh"
stage: source_test
tags:
- arch:amd64
- specific:true
timeout: 2h
variables:
AIX_AGENT_SRC: /opt/dd-build/datadog-agent
AIX_GIT_REMOTE: https://github.com/DataDog/datadog-agent.git
AIX_HOST: cloud3.siteox.com
AIX_PORT: '42222'
AIX_USER: root
GIT_DEPTH: 1aix_unit_testsaix_unit_tests:
before_script:
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- touch ~/.ssh/aix_ci_key && chmod 600 ~/.ssh/aix_ci_key
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SITEOX_AIX_CI_RUNNER key > ~/.ssh/aix_ci_key
|| exit $?
- chmod 600 ~/.ssh/aix_ci_key
- ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null
- 'echo "=== AIX: cloning agent source at $CI_COMMIT_SHA ==="'
- "ssh -T -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST bash -s <<EOF\n\
set -eu\nexport PATH=/opt/freeware/bin:/usr/bin:\\$PATH\nif [ ! -d \"$AIX_AGENT_SRC/.git\"\
\ ]; then\n git clone --progress \"$AIX_GIT_REMOTE\" \"$AIX_AGENT_SRC\"\nfi\n\
cd \"$AIX_AGENT_SRC\"\ngit clean -df\ngit reset --hard\ngit fetch --progress origin\n\
# The commit under test may not be reachable from origin yet (e.g. a\n# brand-new\
\ branch push); fetch the branch ref as a fallback.\ngit fetch --progress origin\
\ \"$CI_COMMIT_REF_NAME\" || true\ngit checkout \"$CI_COMMIT_SHA\"\nEOF\n"
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
needs: []
resource_group: aix_ci_host
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- 'echo "=== AIX: provisioning host (setup-host.sh) ==="'
- ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/setup-host.sh"
- 'echo "=== AIX: running unit tests (run-tests.sh) ==="'
- ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/ci/run-tests.sh"
stage: source_test
tags:
- arch:amd64
- specific:true
timeout: 2h
variables:
AIX_AGENT_SRC: /opt/dd-build/datadog-agent
AIX_GIT_REMOTE: https://github.com/DataDog/datadog-agent.git
AIX_HOST: cloud3.siteox.com
AIX_PORT: '42222'
AIX_USER: root
GIT_DEPTH: 1Changes Summary
ℹ️ Diff available in the job log. |
|
🎯 Code Coverage (details) 🔗 Commit SHA: 9557af8 | Docs | View more details | Give us feedback! |
Files inventory check summaryFile checks results against ancestor 6a44f563: Results for datadog-agent_7.84.0~devel.git.590.9557af8.pipeline.134045802-1_amd64.deb:No change detected Results for datadog-iot-agent_7.84.0~devel.git.590.9557af8.pipeline.134045802-1_amd64.deb:No change detected |
- chmod +x packaging/aix/setup-host.sh and packaging/aix/ci/run-tests.sh (they were committed as 100644, so AIX refused to execute them: 'Execute permission denied' -> exit 1 with no output). - invoke the scripts via 'sh' so the job works even if the executable bit isn't set on the checked-out file. - add -tt to the ssh invocations so output is streamed back to the GitLab job log (without a TTY the remote script's stdout/stderr was swallowed). - switch rules from manual to on_success while the PR is being iterated on.
The clone heredoc ran over SSH without a TTY, so git fetch's progress went to stderr which was swallowed — the job looked frozen for 13+ min at 'git fetch origin' (it was actually working silently). Add -tt to force a PTY (so output streams to the job log) and --progress to the git commands, and set -x so every step is echoed for debuggability.
The -tt flag on the clone heredoc forced bash -s into interactive mode, so after the heredoc script completed bash kept waiting for input and the before_script step hung indefinitely. Use -T (disable TTY) for the heredoc; --progress on the git commands forces progress output to stderr even without a TTY. Keep -tt on the script: steps (single commands that exit cleanly) so setup-host.sh / run-tests.sh output streams live.
Add a trivial smoke test so --only-modified-packages picks up pkg/version and the AIX CI job runs its tests end-to-end.
This reverts commit d05e1b0.
Path-based auto-triggers will be added later once the job is proven stable.
Add echo markers before the clone, setup, and test steps so the job log clearly shows which command is running, alongside the streamed output (-tt / --progress already bring the output through).
The ': ' in '=== AIX: ...' made YAML parse the echo lines as mappings
(key: value) instead of scalar strings, failing the gitlab-configuration
pre-push hook ('before_script config should be a string or a nested array
of strings'). Single-quote the whole line so the colon is literal.
What does this PR do?
Add a manual gitlab CI job to run Go unit tests on an AIX host.
Also add scripts to do the setup of the host.
Motivation
Enable devs to validate their changes on AIX.
Describe how you validated your changes
Manual testing.
Additional Notes