CLOUDP-341025: Refactor of existing snippets on master

[lsierant]

Summary

This PR refactors how we generate and manage outputs from snippet tests.

Before, every snippet module has its own output/ directory. But because snippet modules were reused across different tests we had no way of storing the outputs from different test runs (the files were overwritten).

Changes were needed to accommodate upcoming Search snippets, but due to the large scale of this refactoring it was extracted onto master.

Only changes to public/architectures has been made performed in this PR. Existing search snippets are left intact and will be refactored on the search feature branch. After this PR is approved, a followup PR in the docs repo will be created adjusting for the new include paths.

This PR incorporated changes from #150 to fix diagnostic dumps from GKE clusters.

What this change will help

• committing back snippet output will be easy as adding changes from one directory scripts/code_snippets/tests/outputs/test_* instead of traversing the tree and dealing with the conflicts pushed from different test runs.
• more clarity what snippets and outputs we're using in docs
• ability to share snippet modules and gather different outputs from the same snippet module for different testing scenarios.

Glossary

• Snippet module - directory containing test.sh and scripts in a code_snippets sub directory. It's a reusable set of scripts, e.g. public/architectures/setup-multi-cluster/ra-01-setup-gke.
• Snippet test - automated test running in evg that runs multiple snippet modules. It's a top level entrypoint that runs all the snippet modules. Examples: scripts/code_snippets/tests/test_gke_multi_cluster_no_mesh_snippets.sh,
  scripts/code_snippets/test_gke_multi_cluster_snippets.sh.
  • Previously we named these files a "task" and they were directly in scripts/code_snippets dir (e.g. scripts/code_snippets/task_gke_multi_cluster_no_mesh_snippets_test.sh).
• Snippet outputs - output files (.out) gathered from a Snippet test.

Changes

• Snippet modules don't store their outputs in the snippet directory now, but in scripts/code_snippets/tests/outputs/<test name> directory, which is gathering all outputs from a given snippet test.
• Every file name in any of the code_snippets directories in the codebase must have a globally unique name. Existing file names has been refactored to match the new convention (in public/architecture).
  • Validation/linter has been added (scripts/code_snippets/validate_snippets.py) added to pre-commit that enforces this convention.
• The naming convention is the following:
• Snippet module directory must be prefixed by some arbitrary prefix. Snippet modules in reference architectures has been prefixed with "ra-NN, e.g. ra-02-setup-operator. Every snippet file belonging to this module should be also prefixed (e.g. ra-02_0046_create_image_pull_secrets.sh`)
• This allows for storing all the snippets outputs in a flat directory, guaranteeing we won't overwrite outputs.
• This also helps quickly verifying the paths that will be used in the docs to include snippets and outputs. Having those prefixes will help identifying if any included file is coming from a correct snippet test and module.
• Improved diagnostic gathering from the test runs
  • GKE clusters are correctly handled now
  • Every file name is prefixed with the kube context name and the namespace from which the files were downloaded. This will impact all the e2e test runs, but it will improve diagnostic archive in cases where we dump more than one namespace or k8s cluster.
• All evergreen yamls related to snippets have been moved into .evergreen-snippets.yml (included from the main .evergreen-snippets.yml)

Here is the fragment of the current directory structure:

public/architectures
├── ra-06-ops-manager-multi-cluster
│   ├── code_snippets
│   │   ├── ra-06_0250_generate_certs.sh
│   │   ├── ra-06_0300_ops_manager_create_admin_credentials.sh
            [...]
│   │   ├── ra-06_9100_delete_backup_namespaces.sh
│   │   └── ra-06_9200_delete_om.sh
│   ├── env_variables.sh
│   ├── teardown.sh
│   └── test.sh
├── ra-07-mongodb-replicaset-multi-cluster
│   ├── code_snippets
│   │   ├── ra-07_1050_generate_certs.sh
│   │   ├── ra-07_1100_mongodb_replicaset_multi_cluster.sh
            [...]
│   │   └── ra-07_9000_delete_resources.sh
│   ├── env_variables.sh
│   ├── teardown.sh
│   └── test.sh

scripts/code_snippets
├── tests
│   ├── outputs
│   │   ├── test_gke_multi_cluster_no_mesh_snippets
│   │   ├── test_gke_multi_cluster_snippets
│   │   │   ├── ra-01_0030_verify_access_to_clusters.out
│   │   │   ├── ra-02_0200_kubectl_mongodb_configure_multi_cluster.out
│   │   │   ├── ra-02_0205_helm_configure_repo.out
            [...]
│   │   │   ├── ra-06_0321_ops_manager_wait_for_pending_state.out
│   │   │   ├── ra-06_0322_ops_manager_wait_for_running_state.out
│   │   │   └── ra-06_0522_ops_manager_wait_for_running_state.out
│   │   └── test_kind_search_community_snippets
│   │       ├── 01_0090_helm_add_mogodb_repo.out
│   │       ├── 01_0100_install_operator.out
            [...]
│   │       ├── 03_0450_execute_search_query.out
│   │       └── 03_0455_execute_vector_search_query.out
│   ├── test_gke_multi_cluster_no_mesh_snippets.sh
│   ├── test_gke_multi_cluster_snippets.sh
│   └── test_kind_search_community_snippets.sh

Proof of Work

Green EVG run for all snippets.

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.3.0 Release Notes

New Features

Multi-Architecture Support

We've added comprehensive multi-architecture support for the kubernetes operator. This enhancement enables deployment on IBM Power (ppc64le) and IBM Z (s390x) architectures alongside
existing x86_64 support. Core images (operator, agent, init containers, database, readiness probe) now support multiple architectures. We do not add support IBM and ARM support for Ops-Manager and the init-ops-manager image.

Bug Fixes

• This change fixes the current complex and difficult-to-maintain architecture for stateful set containers, which relies on an "agent matrix" to map operator and agent versions which led to a sheer amount of images.
• We solve this by shifting to a 3-container setup. This new design eliminates the need for the operator-version/agent-version matrix by adding one additional container containing all required binaries. This architecture maps to what we already do with the mongodb-database container.
• Fixed an issue where the readiness probe reported the node as ready even when its authentication mechanism was not in sync with the other nodes, potentially causing premature restarts.

Other Changes

• Optional permissions for PersistentVolumeClaim moved to a separate role. When managing the operator with Helm it is possible to disable permissions for PersistentVolumeClaim resources by setting operator.enablePVCResize value to false (true by default). When enabled, previously these permissions were part of the primary operator role. With this change, permissions have a separate role.
• subresourceEnabled Helm value was removed. This setting used to be true by default and made it possible to exclude subresource permissions from the operator role by specifying false as the value. We are removing this configuration option, making the operator roles always have subresource permissions. This setting was introduced as a temporary solution for this OpenShift issue. The issue has since been resolved and the setting is no longer needed.
• We have deliberately not published the container images for OpsManager versions 7.0.16, 8.0.8, 8.0.9 and 8.0.10 due to a bug in the OpsManager which prevents MCK customers to upgrade their OpsManager deployments to those versions.

[lucian-tosa]

nit: why?

[lsierant]

oh, thanks, it's a debugging leftover!

[lucian-tosa]

nit: This indentation looks off in github.