Skip to content

[WIP] Add a CI job to run unit tests on an AIX host - #55602

Draft
pgimalac wants to merge 14 commits into
mainfrom
pgimalac/aix-ci-unit-tests
Draft

[WIP] Add a CI job to run unit tests on an AIX host#55602
pgimalac wants to merge 14 commits into
mainfrom
pgimalac/aix-ci-unit-tests

Conversation

@pgimalac

Copy link
Copy Markdown
Member

What does this PR do?

Add a manual gitlab CI job to run Go unit tests on an AIX host.
Also add scripts to do the setup of the host.

Motivation

Enable devs to validate their changes on AIX.

Describe how you validated your changes

Manual testing.

Additional Notes

AIX/ppc64 has no Bazel, so  cannot use  to drive unit tests. Mirror the existing
AIX/no-Bazel fallback in tasks/schema/generate.py: on AIX, invoke the
prebuilt gotestsum binary directly via ctx.run and wrap the invoke
Result into a subprocess.CompletedProcess so the rest of the test loop
(which reads res.returncode / res.stdout) is unchanged.

Non-AIX behavior is unchanged.
ctx.run goes through a shell, so building the command by f-string
interpolation (f'gotestsum {formatted_cmd}') let shell metacharacters
in the command — e.g. | in a -run TestA|TestB regex from
--test-run-name — be interpreted by the shell instead of passed
literally to go test.

Build the exact argv (the same shlex.split list the bazel path passes
to gotestsum) and re-quote it with shlex.join, so the shell parses it
back to that same argv. Verified on AIX: --test-run-name='TestA|TestB'
runs the matching tests instead of failing with 'TestB: not found'.
@pgimalac pgimalac added changelog/no-changelog No changelog entry needed qa/no-code-change No code change in Agent code requiring validation labels Aug 28, 2026
@github-actions github-actions Bot added the medium review PR review might take time label Aug 28, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review by Codex (OpenAI) - workflow run

Patch is incorrect: the job executes PR-controlled code as root on a persistent shared host and does not securely authenticate that host.

Comment thread .gitlab/test/aix/aix_unit_tests.yml Outdated
# Setup the host (idempotent) then run the unit tests.
# setup-host.sh is shared with devs (packaging/aix/), run-tests.sh is the
# unit-test job's script (packaging/aix/ci/).
- ssh -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "$AIX_AGENT_SRC/packaging/aix/setup-host.sh"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This executes setup-host.sh from the commit under review as root on a long-lived host shared by later jobs. A malicious or compromised PR can install persistent code, alter the checkout/toolchain, and contaminate subsequent trusted runs. Provision from trusted code outside the PR and run tests as an unprivileged user, or use a fresh isolated host per job.

Comment on lines +38 to +39
# Trust the host key
- ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ssh-keyscan retrieves the key over the same unauthenticated network connection it is meant to secure, so a MITM can substitute its own host key. Store the expected SiteOX host key/fingerprint in a protected CI variable or Vault and verify against that pinned value.

@dd-octo-sts

dd-octo-sts Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Gitlab CI Configuration Changes

Modified Jobs

variables (configuration)
  variables:
    AGENT_API_KEY_ORG2: agent-api-key-org-2
    AGENT_APP_KEY_ORG2: agent-app-key-org-2
    AGENT_BINARIES_DIR: bin/agent
    AGENT_QA_E2E: agent-qa-e2e
    ARTIFACT_DOWNLOAD_ATTEMPTS: 2
    ATLASSIAN_WRITE: atlassian-write
    BTFHUB_ARCHIVE_BRANCH: main
    BUCKET_BRANCH: dev
    CACHE_COMPRESSION_LEVEL: slowest
    CHANGELOG_COMMIT_SHA: ci.datadog-agent.gitlab_changelog_commit_sha
    CI_IMAGE_BTF_GEN: v132333024-4b82e653
    CI_IMAGE_BTF_GEN_SUFFIX: ''
    CI_IMAGE_DOCKER_ARM64: v132333024-4b82e653
    CI_IMAGE_DOCKER_ARM64_SUFFIX: ''
    CI_IMAGE_DOCKER_X64: v132333024-4b82e653
    CI_IMAGE_DOCKER_X64_SUFFIX: ''
    CI_IMAGE_GITLAB_AGENT_DEPLOY: v132333024-4b82e653
    CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX: ''
    CI_IMAGE_LINUX: v132333024-4b82e653
    CI_IMAGE_LINUX_SUFFIX: ''
    CI_IMAGE_RPM_ARM64: v132333024-4b82e653
    CI_IMAGE_RPM_ARM64_SUFFIX: ''
    CI_IMAGE_RPM_ARMHF: v132333024-4b82e653
    CI_IMAGE_RPM_ARMHF_SUFFIX: ''
    CI_IMAGE_RPM_X64: v132333024-4b82e653
    CI_IMAGE_RPM_X64_SUFFIX: ''
    CI_IMAGE_WIN_LTSC2022_X64: v130938967-3a9f81bc
    CI_IMAGE_WIN_LTSC2022_X64_SUFFIX: ''
    CI_IMAGE_WIN_LTSC2025_X64: v130938967-3a9f81bc
    CI_IMAGE_WIN_LTSC2025_X64_SUFFIX: ''
    CLANG_BUILD_VERSION: v60409452-ee70de70
    CLANG_LLVM_VER: 12.0.1
    CLUSTER_AGENT_BINARIES_DIR: bin/datadog-cluster-agent
    CLUSTER_AGENT_CLOUDFOUNDRY_BINARIES_DIR: bin/datadog-cluster-agent-cloudfoundry
    CODECOV: codecov
    COMPARE_TO_BRANCH: main
    CRC_PULL_SECRET: ci.datadog-agent.crc-pull-secret
    CWS_INSTRUMENTATION_BINARIES_DIR: bin/cws-instrumentation
    DATADOG_AGENT_EMBEDDED_PATH: /opt/datadog-agent/embedded
    DDA_CLIENT_TOKEN: dda-feature-flags-client-token
    DDA_FEATURE_FLAGS_CI_SSM_KEY_WINDOWS: ci.datadog-agent.dda-feature-flags-client-token
    DDA_FEATURE_FLAGS_CI_VAULT_KEY: token
    DDA_FEATURE_FLAGS_CI_VAULT_KEY_MACOS: token
    DDA_FEATURE_FLAGS_CI_VAULT_PATH: k8s/gitlab-runner-datadog-agent/datadog-agent/$DDA_CLIENT_TOKEN
    DDA_FEATURE_FLAGS_CI_VAULT_PATH_MACOS: aws/arn:aws:iam::486234852809:role/ci-datadog-agent/$DDA_CLIENT_TOKEN
    DD_AGENT_TESTING_DIR: $CI_PROJECT_DIR/test/new-e2e/tests
    DD_PKG_GITLAB_URL: https://artifact-gateway.us1.ddbuild.io/internal/artifact-gateway/api/v4
    DEB_GPG_KEY_ID: c0962c7d
    DEB_GPG_KEY_NAME: Datadog, Inc. APT key
    DEB_RPM_TESTING_BUCKET_BRANCH: testing
    DEB_S3_BUCKET: apt.datad0g.com
    DEB_TESTING_S3_BUCKET: apttesting.datad0g.com
    DISABLE_GIT_CACHE: true
    DOCKER_REGISTRY_RO: dockerhub-readonly
    DOCKER_REGISTRY_URL: docker.io
    DOGSTATSD_BINARIES_DIR: bin/dogstatsd
    DYNAMIC_TESTS_BREAKGLASS: dynamic-tests-breakglass
    E2E_AZURE: e2e-azure
    E2E_COVERAGE_PIPELINE: false
    E2E_GCP: e2e-gcp
    EXECUTOR_JOB_SECTION_ATTEMPTS: 2
    FF_CLEAN_UP_FAILED_CACHE_EXTRACT: true
    FF_KUBERNETES_HONOR_ENTRYPOINT: true
    FF_SCRIPT_SECTIONS: 1
    FF_TIMESTAMPS: true
    FF_USE_FASTZIP: true
    FF_USE_WINDOWS_JOB_OBJECT: true
    GENERAL_ARTIFACTS_CACHE_BUCKET_URL: https://dd-agent-omnibus.s3.amazonaws.com
    GET_SOURCES_ATTEMPTS: 2
    GIT_DEPTH: 1
    GIT_STRATEGY: fetch
    GO_TEST_SKIP_FLAKE: 'true'
    GPG_TEST_KEY_ID: crypto/k8s/keys/k8s_gitlab-runner-datadog-agent_datadog-agent_testing_signing-key
    INSTALLER_TESTING_S3_BUCKET: installtesting.datad0g.com
    INSTALL_SCRIPT_API_KEY_ORG2: install-script-api-key-org-2
    KERNEL_MATRIX_TESTING_ARM_AMI_ID: ami-0b5f838a19d37fc61
    KERNEL_MATRIX_TESTING_X86_AMI_ID: ami-05b3973acf5422348
    KITCHEN_INFRASTRUCTURE_FLAKES_RETRY: 2
    MACOS_APPLE_APPLICATION_SIGNING: apple-application-signing
    MACOS_APPLE_DEVELOPER_ACCOUNT: apple-developer-account
    MACOS_APPLE_INSTALLER_SIGNING: apple-installer-signing
    MACOS_KEYCHAIN_PWD: ci-keychain
    MACOS_S3_BUCKET: dd-agent-macostesting
    OMNIBUS_BASE_DIR: /omnibus
    OMNIBUS_PACKAGE_DIR: $CI_PROJECT_DIR/omnibus/pkg/
    OMNIBUS_PACKAGE_DIR_SUSE: $CI_PROJECT_DIR/omnibus/suse/pkg
    PIPELINE_KEY_ALIAS: alias/ci_datadog-agent_pipeline-key
    PROCESS_S3_BUCKET: datad0g-process-agent
    PYTHONUNBUFFERED: 1
    RESTORE_CACHE_ATTEMPTS: 2
    RPM_GPG_KEY_ID: b01082d3
    RPM_GPG_KEY_NAME: Datadog, Inc. RPM key
    RPM_S3_BUCKET: yum.datad0g.com
    RPM_TESTING_S3_BUCKET: yumtesting.datad0g.com
    RUN_E2E_TESTS: auto
    RUN_KMT_TESTS: auto
    RUN_UNIT_TESTS: auto
    S3_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    S3_CP_CMD: aws s3 cp $S3_CP_OPTIONS
    S3_CP_OPTIONS: --no-progress --region us-east-1 --sse AES256
    S3_DD_AGENT_OMNIBUS_BTFS_URI: s3://dd-agent-omnibus/btfs
    S3_DD_AGENT_OMNIBUS_JAVA_URI: s3://dd-agent-omnibus/openjdk
    S3_DD_AGENT_OMNIBUS_LLVM_URI: s3://dd-agent-omnibus/llvm
    S3_DSD6_URI: s3://dsd6-staging
    S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
    S3_OMNIBUS_GIT_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-git-cache-build-stable
    S3_PERMANENT_ARTIFACTS_URI: s3://dd-ci-persistent-artefacts-build-stable/$CI_PROJECT_NAME
    S3_PROJECT_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME
    S3_RELEASE_ARTIFACTS_URI: s3://dd-release-artifacts/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    S3_RELEASE_INSTALLER_ARTIFACTS_URI: s3://dd-release-artifacts/datadog-installer/$CI_PIPELINE_ID
    S3_SBOM_STORAGE_URI: s3://sbom-root-us1-ddbuild-io/$CI_PROJECT_NAME/$CI_PIPELINE_ID
    SECRET_GENERIC_CONNECTOR_BINARIES_DIR: bin/secret-generic-connector
+   SITEOX_AIX_CI_RUNNER: siteox-aix-ci-runner
    SKIP_WINDOWS: 'false'
    SLACK_AGENT: slack-agent-ci
    SMP_ACCOUNT: smp
    STATIC_BINARIES_DIR: bin/static
    SYSTEM_PROBE_BINARIES_DIR: bin/system-probe
    TEST_KEYS_URL: apttesting.datad0g.com/test-keys
    VCPKG_BLOB_SAS_URL: vcpkg-blob-sas
    VIRUS_TOTAL: virus-total
    WINDOWS_BUILDS_S3_BUCKET: $WIN_S3_BUCKET/builds
    WINDOWS_SYMBOLS_S3_BUCKET: pipelines/windows-symbols
    WINDOWS_TESTING_S3_BUCKET: pipelines/A7/$CI_PIPELINE_ID
    WINGET_PAT: winget-pat
    WIN_S3_BUCKET: dd-agent-mstesting

Added Jobs

.aix_remote
.aix_remote:
  before_script:
  - mkdir -p ~/.ssh && chmod 700 ~/.ssh
  - touch ~/.ssh/aix_ci_key && chmod 600 ~/.ssh/aix_ci_key
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SITEOX_AIX_CI_RUNNER key > ~/.ssh/aix_ci_key
    || exit $?
  - chmod 600 ~/.ssh/aix_ci_key
  - ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null
  - 'echo "=== AIX: cloning agent source at $CI_COMMIT_SHA ==="'
  - "ssh -T -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST bash -s <<EOF\n\
    set -eu\nexport PATH=/opt/freeware/bin:/usr/bin:\\$PATH\nif [ ! -d \"$AIX_AGENT_SRC/.git\"\
    \ ]; then\n  git clone --progress \"$AIX_GIT_REMOTE\" \"$AIX_AGENT_SRC\"\nfi\n\
    cd \"$AIX_AGENT_SRC\"\ngit clean -df\ngit reset --hard\ngit fetch --progress origin\n\
    # The commit under test may not be reachable from origin yet (e.g. a\n# brand-new\
    \ branch push); fetch the branch ref as a fallback.\ngit fetch --progress origin\
    \ \"$CI_COMMIT_REF_NAME\" || true\ngit checkout \"$CI_COMMIT_SHA\"\nEOF\n"
  image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
  needs: []
  resource_group: aix_ci_host
  rules:
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - allow_failure: true
    when: manual
  script:
  - 'echo "=== AIX: provisioning host (setup-host.sh) ==="'
  - ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/setup-host.sh"
  - 'echo "=== AIX: running unit tests (run-tests.sh) ==="'
  - ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/ci/run-tests.sh"
  stage: source_test
  tags:
  - arch:amd64
  - specific:true
  timeout: 2h
  variables:
    AIX_AGENT_SRC: /opt/dd-build/datadog-agent
    AIX_GIT_REMOTE: https://github.com/DataDog/datadog-agent.git
    AIX_HOST: cloud3.siteox.com
    AIX_PORT: '42222'
    AIX_USER: root
    GIT_DEPTH: 1
aix_unit_tests
aix_unit_tests:
  before_script:
  - mkdir -p ~/.ssh && chmod 700 ~/.ssh
  - touch ~/.ssh/aix_ci_key && chmod 600 ~/.ssh/aix_ci_key
  - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SITEOX_AIX_CI_RUNNER key > ~/.ssh/aix_ci_key
    || exit $?
  - chmod 600 ~/.ssh/aix_ci_key
  - ssh-keyscan -p $AIX_PORT $AIX_HOST >> ~/.ssh/known_hosts 2>/dev/null
  - 'echo "=== AIX: cloning agent source at $CI_COMMIT_SHA ==="'
  - "ssh -T -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST bash -s <<EOF\n\
    set -eu\nexport PATH=/opt/freeware/bin:/usr/bin:\\$PATH\nif [ ! -d \"$AIX_AGENT_SRC/.git\"\
    \ ]; then\n  git clone --progress \"$AIX_GIT_REMOTE\" \"$AIX_AGENT_SRC\"\nfi\n\
    cd \"$AIX_AGENT_SRC\"\ngit clean -df\ngit reset --hard\ngit fetch --progress origin\n\
    # The commit under test may not be reachable from origin yet (e.g. a\n# brand-new\
    \ branch push); fetch the branch ref as a fallback.\ngit fetch --progress origin\
    \ \"$CI_COMMIT_REF_NAME\" || true\ngit checkout \"$CI_COMMIT_SHA\"\nEOF\n"
  image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
  needs: []
  resource_group: aix_ci_host
  rules:
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - allow_failure: true
    when: manual
  script:
  - 'echo "=== AIX: provisioning host (setup-host.sh) ==="'
  - ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/setup-host.sh"
  - 'echo "=== AIX: running unit tests (run-tests.sh) ==="'
  - ssh -tt -i ~/.ssh/aix_ci_key -p $AIX_PORT $AIX_USER@$AIX_HOST "sh $AIX_AGENT_SRC/packaging/aix/ci/run-tests.sh"
  stage: source_test
  tags:
  - arch:amd64
  - specific:true
  timeout: 2h
  variables:
    AIX_AGENT_SRC: /opt/dd-build/datadog-agent
    AIX_GIT_REMOTE: https://github.com/DataDog/datadog-agent.git
    AIX_HOST: cloud3.siteox.com
    AIX_PORT: '42222'
    AIX_USER: root
    GIT_DEPTH: 1

Changes Summary

Removed Modified Added Renamed
0 1 2 0

ℹ️ Diff available in the job log.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 52.73% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9557af8 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Files inventory check summary

File checks results against ancestor 6a44f563:

Results for datadog-agent_7.84.0~devel.git.590.9557af8.pipeline.134045802-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.84.0~devel.git.590.9557af8.pipeline.134045802-1_amd64.deb:

No change detected

- chmod +x packaging/aix/setup-host.sh and packaging/aix/ci/run-tests.sh
  (they were committed as 100644, so AIX refused to execute them: 'Execute
  permission denied' -> exit 1 with no output).
- invoke the scripts via 'sh' so the job works even if the executable bit
  isn't set on the checked-out file.
- add -tt to the ssh invocations so output is streamed back to the GitLab
  job log (without a TTY the remote script's stdout/stderr was swallowed).
- switch rules from manual to on_success while the PR is being iterated on.
The clone heredoc ran over SSH without a TTY, so git fetch's progress went
to stderr which was swallowed — the job looked frozen for 13+ min at
'git fetch origin' (it was actually working silently). Add -tt to force a
PTY (so output streams to the job log) and --progress to the git commands,
and set -x so every step is echoed for debuggability.
The -tt flag on the clone heredoc forced bash -s into interactive mode,
so after the heredoc script completed bash kept waiting for input and the
before_script step hung indefinitely. Use -T (disable TTY) for the
heredoc; --progress on the git commands forces progress output to stderr
even without a TTY. Keep -tt on the script: steps (single commands that
exit cleanly) so setup-host.sh / run-tests.sh output streams live.
Add a trivial smoke test so --only-modified-packages picks up pkg/version
and the AIX CI job runs its tests end-to-end.
Path-based auto-triggers will be added later once the job is proven stable.
Add echo markers before the clone, setup, and test steps so the job log
clearly shows which command is running, alongside the streamed output
(-tt / --progress already bring the output through).
The ': ' in '=== AIX: ...' made YAML parse the echo lines as mappings
(key: value) instead of scalar strings, failing the gitlab-configuration
pre-push hook ('before_script config should be a string or a nested array
of strings'). Single-quote the whole line so the colon is literal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog No changelog entry needed internal Identify a non-fork PR medium review PR review might take time qa/no-code-change No code change in Agent code requiring validation team/agent-build team/agent-devx team/agent-runtimes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant