[CLOUDP-352109] Run MCK E2E tests against OCI published helm chart (#509)

# Summary

As part of our work to move towards OCI compatible container registries
for our helm chart, we are also planning to run our E2E tests against
the helm chart that we publish to OCI. This will make sure that we are
testing in our E2E what we are providing to our customers.
As part of this effort we have already [raised a
PR](#507) that starts
publishing our helm chart to the OCI container registry during
dev/staging workflows.
This PR goes and changes our E2E tests to start consuming the helm chart
from OCI registry instead of the local helm chart repo.

Additional unrelated change is to fix `kubectl-mongodb` location that is
used for gke code snippets. The fix is to add missing `workdir` path
suffix.

## Proof of Work

Successful CI on this PR.
I also ran the test `e2e_replica_set_migration` locally and it was
successful. The logs are
[here](https://gist.github.com/viveksinghggits/1b6403ffd43d53759a1714aa9cf04f30).

`publish_helm_chart` example output:
```
[2025/11/05 22:11:19.857] Running command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7).
[2025/11/05 22:11:19.919] INFO     2025-11-05 21:11:19,919 [publish_helm_chart]  Packaging chart: mongodb-kubernetes with Version: 0.0.0+690bbc0f836fbf0007154375
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully executed: helm package --version 0.0.0+690bbc0f836fbf0007154375 helm_chart
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully packaged chart and saved it to: /data/mci/55da5d1be5488b41a9ec4e7d02041a86/src/github.com/mongodb/mongodb-kubernetes/mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Determined OCI Registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Pushing chart to registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Successfully executed: helm push mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Helm Chart mongodb-kubernetes:0.0.0+690bbc0f836fbf0007154375 was published successfully!
[2025/11/05 22:11:20.858] Finished command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7) in 1.001053371s.
```

Downloading the chart with `helm pull
oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts/mongodb-kubernetes
--untar --version 0.0.0+690bbc0f836fbf0007154375` you can see that the
`Chart.yaml` version was properly updated:
```yaml
apiVersion: v2
description: 'MongoDB Controllers for Kubernetes translate the human knowledge of
  creating a MongoDB instance into a scalable, repeatable, and standardized method. '
home: https://github.com/mongodb/mongodb-kubernetes
icon: https://mongodb-images-new.s3.eu-west-1.amazonaws.com/leaf-green-dark.png
keywords:
  - mongodb
  - database
  - nosql
kubeVersion: '>=1.16-0'
maintainers:
  - email: [email protected]
    name: MongoDB
name: mongodb-kubernetes
type: application
version: 0.0.0+690bbc0f836fbf0007154375
```

Also [run
release](https://evergreen.mongodb.com/version/690cf486c59aec0007dd2ba4?redirect_spruce_users=true)
test and e2e smoke tests are also passing (ignore `ibm_power`, they are
fixed in #573)
## Checklist

- [x] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

---------

Co-authored-by: Maciej Karaś <[email protected]>

Author: viveksinghggits

.evergreen-release.yml

@@ -214,6 +214,8 @@ buildvariants:
     depends_on:
       - name: release_kubectl_mongodb_plugin
         variant: release_kubectl_mongodb_plugin
+      - name: release_chart_to_oci_registry
+        variant: release_chart_to_oci_registry
     tasks:
       - name: build_test_image
 
@@ -228,6 +230,8 @@ buildvariants:
     depends_on:
       - name: release_kubectl_mongodb_plugin
         variant: release_kubectl_mongodb_plugin
+      - name: release_chart_to_oci_registry
+        variant: release_chart_to_oci_registry
     tasks:
       - name: build_test_image_ibm_power
 
@@ -245,6 +249,8 @@ buildvariants:
     depends_on:
       - name: release_kubectl_mongodb_plugin
         variant: release_kubectl_mongodb_plugin
+      - name: release_chart_to_oci_registry
+        variant: release_chart_to_oci_registry
     tasks:
       - name: build_test_image_ibm_z
 
@@ -258,6 +264,8 @@ buildvariants:
     depends_on:
       - name: release_kubectl_mongodb_plugin
         variant: release_kubectl_mongodb_plugin
+      - name: release_chart_to_oci_registry
+        variant: release_chart_to_oci_registry
     tasks:
       - name: build_test_image_arm
 

.evergreen.yml

@@ -37,6 +37,8 @@ variables:
         variant: init_test_run
       - name: build_init_om_images_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
   - &base_no_om_image_dependency
     depends_on:
@@ -54,6 +56,8 @@ variables:
         variant: init_test_run
       - name: build_init_appdb_images_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
   - &community_dependency
     depends_on:
@@ -67,6 +71,8 @@ variables:
         variant: init_test_run
       - name: build_mco_test_image
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
   - &setup_group
     setup_group_can_fail_task: true
@@ -138,6 +144,8 @@ variables:
         variant: init_test_run
       - name: build_init_om_images_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
   - &base_om7_dependency_with_race
     depends_on:
@@ -155,6 +163,8 @@ variables:
         variant: init_test_run
       - name: build_init_om_images_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
   # Any change to base_om8_dependency should be reflected to its copy in .evergreen-snippets.yml
   - &base_om8_dependency
@@ -173,6 +183,8 @@ variables:
         variant: init_test_run
       - name: build_init_om_images_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
 parameters:
   - key: evergreen_retry
@@ -1305,6 +1317,8 @@ buildvariants:
         variant: init_test_run
       - name: build_init_database_image_ubi
         variant: init_test_run
+      - name: publish_helm_chart
+        variant: init_test_run
 
     tasks:
       - name: e2e_custom_domain_task_group
@@ -1442,6 +1456,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_ibm_power
         variant: init_test_run_ibm_power
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_ibm_task_group
 
@@ -1461,6 +1477,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_ibm_power
         variant: init_test_run_ibm_power
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_ibm_task_group
 
@@ -1483,6 +1501,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_ibm_z
         variant: init_test_run_ibm_z
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_ibm_task_group
 
@@ -1503,6 +1523,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_ibm_z
         variant: init_test_run_ibm_z
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_ibm_task_group
 
@@ -1521,6 +1543,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_arm
         variant: init_test_run_arm
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_arm_task_group
 
@@ -1537,6 +1561,8 @@ buildvariants:
         variant: init_test_run
       - name: build_test_image_arm
         variant: init_test_run_arm
+      - name: publish_helm_chart
+        variant: init_test_run
     tasks:
       - name: e2e_smoke_arm_task_group
 

build_info.json

@@ -453,12 +453,14 @@
   "helm-charts": {
     "mongodb-kubernetes": {
       "patch": {
+        "version-prefix": "0.0.0+",
         "registry": "268558157000.dkr.ecr.us-east-1.amazonaws.com",
         "region": "us-east-1",
         "repository": "dev/mongodb/helm-charts"
       },
       "staging": {
         "sign": true,
+        "version-prefix": "0.0.0+",
         "registry": "268558157000.dkr.ecr.us-east-1.amazonaws.com",
         "region": "us-east-1",
         "repository": "staging/mongodb/helm-charts"

docker/mongodb-kubernetes-tests/Dockerfile

@@ -16,7 +16,12 @@ RUN apt-get -qq update \
 
 COPY requirements.txt requirements.txt
 
-RUN python3 -m venv /venv && . /venv/bin/activate && pip install --upgrade pip && GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip install -r requirements.txt
+# install aws, required to run helm registry login while running the tests
+RUN python3 -m venv /venv  \
+    && . /venv/bin/activate  \
+    && pip install --upgrade pip  \
+    && GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip install -r requirements.txt \
+    && pip install awscli
 
 FROM scratch AS tools_downloader
 
@@ -75,8 +80,9 @@ WORKDIR /tests
 
 # copying the test files after python build, otherwise pip install will be called each time the tests change
 COPY . /tests
-# copying the helm_chart directory as well to support installation of the Operator from the test application
-COPY helm_chart /helm_chart
+# copying the helm_chart/crds directory so that we can install CRDs before installing the operator helm chart to run a test.
+# operator is installed via published OCI helm repo and not the local helm repo.
+COPY helm_chart/crds /helm_chart/crds
 COPY release.json /release.json
 # we use the public directory to automatically test resources samples
 COPY public /mongodb-kubernetes/public

docker/mongodb-kubernetes-tests/kubetester/helm.py

@@ -1,15 +1,24 @@
 import glob
-import logging
 import os
 import re
 import subprocess
 import uuid
 from typing import Dict, List, Optional, Tuple
 
 from tests import test_logger
+from tests.constants import (
+    DEFAULT_HELM_CHART_PATH_ENV_VAR_NAME,
+    OCI_HELM_REGION_ENV_VAR_NAME,
+    OCI_HELM_REGISTRY_ENV_VAR_NAME,
+    OCI_HELM_VERSION_ENV_VAR_NAME,
+)
 
 logger = test_logger.get_test_logger(__name__)
 
+# LOCAL_CRDS_DIR is the dir where local helm chart's CRDs are copied in tests image
+LOCAL_CRDS_DIR = "helm_chart/crds"
+OCI_HELM_REGISTRY_ECR = "268558157000.dkr.ecr.us-east-1.amazonaws.com"
+
 
 def helm_template(
     helm_args: Dict,
@@ -25,7 +34,7 @@ def helm_template(
         command_args.append("--show-only")
         command_args.append(templates)
 
-    args = ("helm", "template", *(command_args), helm_chart_path)
+    args = ("helm", "template", *command_args, helm_chart_path)
     logger.info(" ".join(args))
 
     yaml_file_name = "{}.yaml".format(str(uuid.uuid4()))
@@ -145,6 +154,55 @@ def process_run_and_check(args, **kwargs):
         raise
 
 
+def helm_registry_login_to_ecr(helm_registry: str, region: str):
+    logger.info(f"Attempting to log into ECR registry: {helm_registry}, using helm registry login.")
+
+    aws_command = ["aws", "ecr", "get-login-password", "--region", region]
+
+    # as we can see the password is being provided by stdin, that would mean we will have to
+    # pipe the aws_command (it figures out the password) into helm_command.
+    helm_command = ["helm", "registry", "login", "--username", "AWS", "--password-stdin", helm_registry]
+
+    try:
+        logger.info("Starting AWS ECR credential retrieval.")
+        aws_proc = subprocess.Popen(
+            aws_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True  # Treat input/output as text strings
+        )
+
+        logger.info("Starting Helm registry login.")
+        helm_proc = subprocess.Popen(
+            helm_command, stdin=aws_proc.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
+        )
+
+        # Close the stdout stream of aws_proc in the parent process
+        # to prevent resource leakage (only needed if you plan to do more processing)
+        aws_proc.stdout.close()
+
+        # Wait for the Helm command (helm_proc) to finish and capture its output
+        helm_stdout, helm_stderr = helm_proc.communicate()
+
+        # Wait for the AWS process to finish as well
+        aws_proc.wait()
+
+        if aws_proc.returncode != 0:
+            _, aws_stderr = aws_proc.communicate()
+            raise Exception(f"aws command to get password failed. Error: {aws_stderr}")
+
+        if helm_proc.returncode == 0:
+            logger.info("Login to helm registry was successful.")
+            logger.info(helm_stdout.strip())
+        else:
+            raise Exception(
+                f"Login to helm registry failed, Exit code: {helm_proc.returncode}, Error: {helm_stderr.strip()}"
+            )
+
+    except FileNotFoundError as e:
+        # This catches errors if 'aws' or 'helm' are not in the PATH
+        raise Exception(f"Command not found. Please ensure '{e.filename}' is installed and in your system's PATH.")
+    except Exception as e:
+        raise Exception(f"An unexpected error occurred: {e}.")
+
+
 def helm_upgrade(
     name: str,
     namespace: str,
@@ -162,7 +220,7 @@ def helm_upgrade(
     chart_dir = helm_chart_path
 
     if apply_crds_first:
-        apply_crds_from_chart(chart_dir)
+        apply_crds_from_chart(LOCAL_CRDS_DIR)
 
     command_args = _create_helm_args(helm_args, helm_options)
     args = [
@@ -183,11 +241,11 @@ def helm_upgrade(
     process_run_and_check(command, check=True, capture_output=True, shell=True)
 
 
-def apply_crds_from_chart(chart_dir: str):
-    crd_files = glob.glob(os.path.join(chart_dir, "crds", "*.yaml"))
+def apply_crds_from_chart(crds_dir: str):
+    crd_files = glob.glob(os.path.join(crds_dir, "*.yaml"))
 
     if not crd_files:
-        raise Exception(f"No CRD files found in chart directory: {chart_dir}")
+        raise Exception(f"No CRD files found in chart directory: {crds_dir}")
 
     logger.info(f"Found {len(crd_files)} CRD files to apply:")
 
@@ -230,3 +288,29 @@ def _create_helm_args(helm_args: Dict[str, str], helm_options: Optional[List[str
         command_args.extend(helm_options)
 
     return command_args
+
+
+# helm_chart_path_and_version returns the chart path and version that we would like to install to run the E2E tests.
+# for local tests it returns early with local helm chart dir and for other scenarios it figures out the chart and version
+# based on the caller. In most of the cases we will install chart from OCI registry but for the tests where we would like
+# to install MEKO's specific version or MCK's specific version, we would expect `helm_chart_path` to set already.
+def helm_chart_path_and_version(helm_chart_path: str, operator_version: str) -> tuple[str, str]:
+    # if helm_chart_path is not passed, use the default value from DEFAULT_HELM_CHART_PATH
+    if not helm_chart_path:
+        helm_chart_path = os.getenv(DEFAULT_HELM_CHART_PATH_ENV_VAR_NAME)
+
+    if helm_chart_path.startswith("oci://"):
+        # If operator_version is not passed, we want to install the current version.
+        if not operator_version:
+            operator_version = os.environ.get(OCI_HELM_VERSION_ENV_VAR_NAME)
+
+        registry = os.environ.get(OCI_HELM_REGISTRY_ENV_VAR_NAME)
+        # If ECR we need to login first to the OCI container registry
+        if registry == OCI_HELM_REGISTRY_ECR:
+            region = os.environ.get(OCI_HELM_REGION_ENV_VAR_NAME)
+            try:
+                helm_registry_login_to_ecr(registry, region)
+            except Exception as e:
+                raise Exception(f"Failed to login to ECR helm registry {registry}. Error: {e}")
+
+    return helm_chart_path, operator_version