CLOUDP-377831: Rewrite prepare local e2e run script by Julien-Ben · Pull Request #773 · mongodb/mongodb-kubernetes

Julien-Ben · 2026-02-11T13:16:28Z

Summary

The local e2e preparation script (prepare_local_e2e_run.sh) was taking ~45-51s on multi-cluster setups, mostly due to sequential kubectl subprocess spawning. This PR brings it down to ~15-19s.

Every local e2e run starts with this script, so saving 30s in each iteration adds up fast during development.

What changed

Go rewrite of multi-cluster prep: The biggest win. Replaced the bash functions prepare_multi_cluster_e2e_run and configure_multi_cluster_environment with a Go binary (scripts/dev/prepare-multi-cluster/main.go) that uses client-go directly instead of shelling out to kubectl. Operations across clusters run in parallel via goroutines. This alone took the multi-cluster prep step from ~11s to ~1.4s.

The bash version used a helm chart to template RBAC resources (a ServiceAccount and a ClusterRoleBinding). The Go version creates these directly via the typed API with idempotent get-or-create-or-update logic. The helm chart is unchanged and still used by CI; only the local dev path bypasses it.

Bash parallelization: make install and delete_om_projects.sh are backgrounded early in the script and waited on before the deploy step.

Redundant work elimination: create_image_registries_secret was called twice per run (once in configure_container_auth.sh, again in configure_operator.sh). Gated the second call behind RUNNING_IN_EVG so it only runs in CI. Similarly, my-project/my-credentials creation is skipped in local multi-cluster runs since the Go binary handles it.

Smaller wins: kubectl delete + create converted to kubectl apply everywhere. Makefile sentinel file skips CRD regeneration when api/*.go hasn't changed. go build -o bin/X instead of go run (build cache makes repeated runs instant).

CI impact

All changes are gated to only affect local runs. CI paths (e2e.sh, single_e2e.sh) are unchanged in behavior — RUNNING_IN_EVG guards ensure CI still creates pull secrets and config resources through the existing shell path. The Go binary is only invoked from prepare_local_e2e_run.sh.

Follow-ups

Two more changes can be considered to further reduce the total time (by ~3-5s):

Run the kubectl plugin multi cluster setup concurrently (via e.g a --parallel flag).
Further parallelize reset.go (not only across clusters, but also within each cluster).

Proof of Work

Manually ran different e2e tests in different variants to assert correctness. This script doesn't run in CI.

Performance

Ran locally against a 4-cluster KIND setup (1 central + 3 members), where the changes matter most. Timed 3 runs of each version, both from a cold start (no compilation cache, CRDs need regenerating) and warm (caches populated).

                     Cold start     Warm start
                     ----------     ----------
Before (sequential)  51.06s         44.77s
After  (all changes) 19.43s         15.36s

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?

github-actions · 2026-02-11T13:17:30Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.7.1 Release Notes

Other Changes

Container images: Merged the init-database and init-appdb init container images into a single init-database image. The init-appdb image will no longer be published and does not affect existing deployments.

…/github.com/mongodb/mongodb-kubernetes into CLOUDP-377831_rewrite-prepare-local-e2e-run

lsierant · 2026-02-13T20:27:23Z

scripts/dev/prepare-multi-cluster/main.go

+  mtls:
+    mode: STRICT`, namespace)
+
+	cmd := exec.Command("kubectl", "--context", cluster, "-n", namespace, "apply", "-f", "-")


what the hack! 😅
let's use GroupVersionKind and Unstructured data with the yaml payload, essentially mimic kubectl apply but with the API server. We don't need to fallback to kubectl

Oh, I did it initially to to avoid initializing a second (untyped) client, out of laziness.

But here is the fix: 318e91e

nammn · 2026-02-17T08:00:57Z

scripts/funcs/multicluster

+  orgId: "${OM_ORGID:-}"
+EOF
+
+  # Note: create_image_registries_secret is already called by configure_operator.sh


do you think this comment is useful in a later time, assuming this is getting merges?

nammn · 2026-02-17T08:04:42Z

scripts/evergreen/e2e/configure_operator.sh

-# We always create the image pull secret from the docker config.json which gives access to all necessary image repositories
-create_image_registries_secret
+# In local runs, configure_container_auth.sh already creates the pull secret in all clusters
+# before this script runs. Skip the redundant call to avoid a ~0.9s delete+recreate cycle


i am not sure whether those comments make sense here rather than in the pr as inline comments. All of those changes are tight to prior implementation

nammn · 2026-02-17T08:05:37Z

scripts/evergreen/e2e/configure_operator.sh

+# across all clusters. In CI (Evergreen), this script is the only place the pull secret is
+# created (e2e.sh sources it directly without configure_container_auth.sh).
+if [[ "${RUNNING_IN_EVG:-false}" == "true" ]]; then
+  create_image_registries_secret


can't we rather update create_image_registries_secret to skip if the secret already exists? That removes the running in evg check

nammn · 2026-02-17T08:06:54Z

scripts/dev/prepare-multi-cluster/main.go

+}
+
+// Phase 7: Copy or build kubectl-mongodb binary
+func setupKubectlMongodb(cfg config) error {


nit: I think we could split the function into 2 subfunctions for readability:

checkExists()

buildFromSource()

basically your comments but as subfunctions.

nammn · 2026-02-17T08:08:15Z

scripts/dev/prepare-multi-cluster/main.go

+	}
+
+	// Write central_cluster name
+	writeConfigFile(cfg.multiClusterConfigDir, "central_cluster", secret.Data["central_cluster"], collectError)


where do we write to here? Do we even save anything in doing this concurrently?

Julien-Ben added 9 commits February 9, 2026 17:28

Background operations

d3b2160

Prepare multi cluster rewrite to go

c19c2c9

Optimize CRD generation

19bec2e

Idempotent apply

a15f677

Optimize configure operator

7885598

Build go binaries

4d027fd

Move scripts in their own directory

c2b542d

Small fixes

d7a0a11

Lint

38125ad

Julien-Ben added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Feb 11, 2026

Merge branch 'master' into CLOUDP-377831_rewrite-prepare-local-e2e-run

f32ad89

Julien-Ben requested a review from lsierant February 11, 2026 13:19

Julien-Ben and others added 3 commits February 11, 2026 14:36

Fix ctx timeout, use constants for strings

a82d3b7

Merge branch 'CLOUDP-377831_rewrite-prepare-local-e2e-run' of https:/…

d6bb6c2

…/github.com/mongodb/mongodb-kubernetes into CLOUDP-377831_rewrite-prepare-local-e2e-run

Merge branch 'master' into CLOUDP-377831_rewrite-prepare-local-e2e-run

1cbf2c4

Julien-Ben marked this pull request as ready for review February 13, 2026 11:09

Julien-Ben requested a review from a team as a code owner February 13, 2026 11:09

Julien-Ben requested review from filipcirtog and removed request for lsierant February 13, 2026 11:09

Julien-Ben marked this pull request as draft February 13, 2026 11:10

Julien-Ben marked this pull request as ready for review February 13, 2026 11:10

Julien-Ben requested review from lsierant and nammn February 13, 2026 11:10

lsierant reviewed Feb 13, 2026

View reviewed changes

Julien-Ben and others added 2 commits February 16, 2026 13:23

Use dynamic go client for applyPeerAuthentication instead of kubectl

318e91e

Merge branch 'master' into CLOUDP-377831_rewrite-prepare-local-e2e-run

206d113

nammn approved these changes Feb 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLOUDP-377831: Rewrite prepare local e2e run script#773

CLOUDP-377831: Rewrite prepare local e2e run script#773
Julien-Ben wants to merge 15 commits intomasterfrom
CLOUDP-377831_rewrite-prepare-local-e2e-run

Julien-Ben commented Feb 11, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 11, 2026 •

edited

Loading

Uh oh!

lsierant Feb 13, 2026

Uh oh!

Julien-Ben Feb 16, 2026 •

edited

Loading

Uh oh!

nammn Feb 17, 2026

Uh oh!

nammn Feb 17, 2026

Uh oh!

nammn Feb 17, 2026

Uh oh!

nammn Feb 17, 2026

Uh oh!

nammn Feb 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Julien-Ben commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

CI impact

Follow-ups

Proof of Work

Performance

Checklist

Uh oh!

github-actions bot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.7.1 Release Notes

Other Changes

Uh oh!

lsierant Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Julien-Ben Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nammn Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

nammn Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

nammn Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

nammn Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

nammn Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Julien-Ben commented Feb 11, 2026 •

edited

Loading

github-actions bot commented Feb 11, 2026 •

edited

Loading

Julien-Ben Feb 16, 2026 •

edited

Loading