Skip to content

Conversation

avi-biton
Copy link
Contributor

@avi-biton avi-biton marked this pull request as draft September 8, 2025 09:28
@openshift-ci openshift-ci bot requested review from arewm and Roming22 September 8, 2025 09:28
Copy link
Contributor

github-actions bot commented Sep 8, 2025

Code Review by Gemini

## Code Review

The changes introduce `squid` and `trust-manager` applications, update several component versions, modify Kueue configurations, and adjust banner content.

### Potential Issues and Suggestions:

1.  **Squid ApplicationSet `environment` Mismatch**
    *   **Issue**: The `squid.yaml` ApplicationSet explicitly sets the `environment` to `development`. However, the commit message states that Squid should be deployed to "development and staging environments". This configuration will only deploy to development.
    *   **File**: `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml`
    *   **Suggestion**: To deploy to both development and staging, the `environment` value should be dynamic, typically derived from the cluster's environment label or an overlay. A common pattern is to use a `values.environment` placeholder that is then patched by environment-specific overlays.

    ```diff
    --- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    @@ -13,7 +13,7 @@
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
-                environment: development
+                environment: '{{environment}}' # Or a more specific dynamic value
                 clusterDir: ""
           - list:
               elements: []
    ```
    And then ensure the `development` and `staging` overlays patch this `environment` value accordingly.

2.  **Trust Manager ApplicationSet `environment` Mismatch**
    *   **Issue**: The `trust-manager.yaml` ApplicationSet sets `environment: staging`. However, the `argo-cd-apps/overlays/development/kustomization.yaml` file includes a patch for `trust-manager`, indicating it should also be deployed in the `development` environment. With `environment: staging` hardcoded, the development deployment will incorrectly source its configuration from `components/trust-manager/staging/`.
    *   **File**: `argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml`
    *   **Suggestion**: Similar to Squid, the `environment` value should be dynamic.

    ```diff
    --- a/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    @@ -13,7 +13,7 @@
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/trust-manager
-                environment: staging
+                environment: '{{environment}}' # Or a more specific dynamic value
                 clusterDir: ""
           - list:
               elements: []
    ```
    And then ensure the `development` and `staging` overlays patch this `environment` value accordingly.

3.  **Squid Helm Generator `environment` Mismatch**
    *   **Issue**: Both `development/squid-helm-generator.yaml` and `staging/squid-helm-generator.yaml` set `valuesInline.environment: release`. This means the Squid Helm chart will always use "release" specific configurations, even when deployed in development or staging. This is likely unintended.
    *   **File**: `components/squid/development/squid-helm-generator.yaml`
    *   **Suggestion**: The `environment` value should reflect the actual environment.

    ```diff
    --- a/components/squid/development/squid-helm-generator.yaml
    +++ b/components/squid/development/squid-helm-generator.yaml
    @@ -13,4 +13,4 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
    ```

    *   **File**: `components/squid/staging/squid-helm-generator.yaml`
    *   **Suggestion**: The `environment` value should reflect the actual environment.

    ```diff
    --- a/components/squid/staging/squid-helm-generator.yaml
    +++ b/components/squid/staging/squid-helm-generator.yaml
    @@ -13,4 +13,4 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
    ```

4.  **Kueue Test Coverage Regression**
    *   **Issue**: The `hack/test-tekton-kueue-config.py` script has been modified to remove testing of `production-kflux-ocp-p01` and now *only* tests the `development` Kueue configuration. This significantly reduces test coverage for staging and production Kueue configurations, which might have different CEL expressions or priorities.
    *   **File**: `hack/test-tekton-kueue-config.py`
    *   **Suggestion**: Reintroduce testing for staging and production Kueue configurations. The script should be able to dynamically load and test different `config.yaml` files (e.g., `components/kueue/staging/base/tekton-kueue/config.yaml` and `components/kueue/production/base/tekton-kueue/config.yaml`) to ensure comprehensive coverage.

    ```diff
    --- a/hack/test-tekton-kueue-config.py
    +++ b/hack/test-tekton-kueue-config.py
    @@ -1,101 +1,89 @@
     #!/usr/bin/env python3
     """
     Tekton-Kueue Configuration Test
     
     A comprehensive test suite that validates the CEL expressions in the tekton-kueue configuration by:
     
    -1. **Reading configuration dynamically** from specified config files
    -2. **Getting the image** from specified kustomization files
    +1. **Reading configuration dynamically** from `components/kueue/development/tekton-kueue/config.yaml`
    +2. **Getting the image** from `components/kueue/staging/base/tekton-kueue/kustomization.yaml`
     3. **Running mutations** using the actual tekton-kueue container via podman
     4. **Validating results** against expected annotations, labels, and priority classes
     
     Usage:
         # Check if all prerequisites are met
         python hack/test-tekton-kueue-config.py --check-setup
     
         # Run all tests
         python hack/test-tekton-kueue-config.py
     
         # Run tests with verbose output
         python hack/test-tekton-kueue-config.py --verbose
     
     Test Scenarios:
    -    Each test can now specify its own config file and kustomization file,
    -    allowing testing of multiple configurations and images.
    +    The test covers all CEL expressions in the configuration:
    +
    +    1. **Multi-platform Resource Requests**:
    +       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-*` annotations
    +       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-*` annotations
    +
    +    2. **AWS IP Resource Requests**:
    +       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-aws-ip` annotations
    +         for platforms NOT in the excluded list (linux/ppc64le, linux/s390x, linux-x86-64, local, localhost, linux/amd64)
    +       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-aws-ip` annotations
    +         for platforms NOT in the excluded list
+
+    3. **Priority Assignment Logic**:
+       - Push events → `konflux-post-merge-build`
+       - Pull requests → `konflux-pre-merge-build`
+       - Integration test push → `konflux-post-merge-test`
+       - Integration test PR → `konflux-pre-merge-test`
+       - Release managed → `konflux-release`
+       - Release tenant → `konflux-tenant-release`
+       - Mintmaker namespace → `konflux-dependency-update`
+       - Default → `konflux-default`
+
+    4. **Queue Assignment**: All PipelineRuns get `kueue.x-k8s.io/queue-name: pipelines-queue`
     
     Prerequisites:
         - Python 3 with PyYAML
         - Podman (for running the tekton-kueue container)
    -    - Access to the tekton-kueue images specified in the kustomizations
    +    - Access to the tekton-kueue image specified in the kustomization
     
     CI/CD Integration:
         The test runs automatically on pull requests via the GitHub action
         `.github/workflows/test-tekton-kueue-config.yaml` when:
         - Changes are made to `components/kueue/**`
         - The test script itself is modified
         - The workflow file is modified
     
         The test will **FAIL** (not skip) if any prerequisites are missing, ensuring
         issues are caught early in CI/CD pipelines.
     """
     
     import subprocess
     import tempfile
     import os
     import yaml
     import unittest
     from pathlib import Path
    -from typing import Dict, Any, TypedDict
    +from typing import Dict
     from dataclasses import dataclass
     import sys
     
     
     @dataclass
    -class TestConfig:
    +class Prerequisites:
+        image: str
+        podman_version: str
+        config_file: Path
+        kustomization_file: Path
    -    config_file: Path
    -    kustomization_file: Path
    -    image: str
    -
    -
    -class ConfigCombination(TypedDict):
    -    config_file: str
    -    kustomization_file: str
    -
    -
    -class TestCombination(TypedDict):
    -    pipelinerun_key: str
    -    config_key: str
    -
    -
    -class PipelineRunMetadata(TypedDict, total=False):
    -    name: str
    -    namespace: str
    -    labels: Dict[str, str]
    -    annotations: Dict[str, str]
    -
    -
    -class PipelineRunDefinition(TypedDict):
    -    apiVersion: str
    -    kind: str
    -    metadata: PipelineRunMetadata
    -    spec: Dict[str, Any]  # More flexible since PipelineRun specs can vary
    -
    -
    -class ExpectedResults(TypedDict):
    -    annotations: Dict[str, str]
    -    labels: Dict[str, str]
    -
    -
    -class PipelineRunTestData(TypedDict):
    -    pipelinerun: PipelineRunDefinition
    -    expected: ExpectedResults
     
     
     def get_tekton_kueue_image(kustomization_file: Path) -> str:
         """Read the tekton-kueue image from the given kustomization file."""
         try:
             with open(kustomization_file, 'r') as f:
                 kustomization = yaml.safe_load(f)
     
             # Look for the tekton-kueue image in the images section
             images = kustomization.get('images', [])
             for image in images:
                 if image.get('name') == 'tekton-kueue':
                     new_name = image.get('newName', '')
                     new_tag = image.get('newTag', '')
                     if new_name and new_tag:
                         return f"{new_name}:{new_tag}"
     
             raise ValueError("tekton-kueue image not found in kustomization")
     
         except Exception as e:
             raise RuntimeError(f"Failed to read tekton-kueue image from {kustomization_file}: {e}")
     
    -def resolve_path(path_str: str, repo_root: Path) -> Path:
    -    """Resolve a path string to an absolute Path, handling both relative and absolute paths."""
    -    if Path(path_str).is_absolute():
    -        return Path(path_str)
    -    return repo_root / path_str
    -
    -
    -def validate_config_combination(config_key: str, repo_root: Path) -> TestConfig:
    -    """Validate and resolve config and kustomization files for a config combination."""
    -    config_data = CONFIG_COMBINATIONS[config_key]
    -
    -    config_file = resolve_path(config_data["config_file"], repo_root)
    -    kustomization_file = resolve_path(config_data["kustomization_file"], repo_root)
    -
    -    # Validate files exist
    -    if not config_file.exists():
    -        raise FileNotFoundError(f"Config file not found for config '{config_key}': {config_file}")
    -
    -    if not kustomization_file.exists():
    -        raise FileNotFoundError(f"Kustomization file not found for config '{config_key}': {kustomization_file}")
    -
    -    # Get image from kustomization
    -    image = get_tekton_kueue_image(kustomization_file)
    -
    -    return TestConfig(
    -        config_file=config_file,
    -        kustomization_file=kustomization_file,
    -        image=image
    -    )
    -
    -
     def check_prerequisites(should_print: bool = True) -> Prerequisites:
         """Check that all prerequisites are available.
     
         Returns a Prerequisites object with discovered info (image, podman_version)
         on success. Raises an exception on failure.
         """
         messages = ["Checking prerequisites..."]
     
         # Compute repo paths locally
         repo_root = Path(__file__).parent.parent
    -    config_file = repo_root / "components/kueue/development/tekton-kueue/config.yaml"
    -    kustomization_file = repo_root / "components/kueue/development/tekton-kueue/kustomization.yaml"
    +    # The test script should be updated to test all relevant configurations,
    +    # or at least the base ones for staging and production.
    +    # For now, we'll use the development config as a default for the test setup.
    +    config_file = repo_root / "components/kueue/development/tekton-kueue/config.yaml" # This should be dynamic
    +    kustomization_file = repo_root / "components/kueue/development/tekton-kueue/kustomization.yaml" # This should be dynamic
     
         # Config file
         if not config_file.exists():
             raise FileNotFoundError(f"Config file not found: {config_file}")
         messages.append(f"\u2713 Config file found: {config_file}")
     
         # Kustomization file
         if not kustomization_file.exists():
             raise FileNotFoundError(f"Kustomization file not found: {kustomization_file}")
         messages.append(f"\u2713 Kustomization file found: {kustomization_file}")
     
         # Image from kustomization
         image = get_tekton_kueue_image(kustomization_file)
         messages.append(f"\u2713 Tekton-kueue image: {image}")
     
         # Podman availability
         result = subprocess.run(["podman", "--version"], capture_output=True, check=True, text=True)
         podman_version = result.stdout.strip()
         messages.append(f"\u2713 Podman available: {podman_version}")
     
         if should_print:
             for line in messages:
                 print(line)
     
         return Prerequisites(
             image=image,
             podman_version=podman_version,
             config_file=config_file,
             kustomization_file=kustomization_file,
         )
     
    -# Test PipelineRun definitions (reusable across different configs)
    -PIPELINERUN_DEFINITIONS: Dict[str, PipelineRunTestData] = {
    +# Test PipelineRun definitions
    +TEST_PIPELINERUNS = {
         "multiplatform_new": {
             "name": "Multi-platform pipeline (new style with build-platforms parameter)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-multiplatform-new",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
         },
     
         "multiplatform_old": {
             "name": "Multi-platform pipeline (old style with PLATFORM parameter in tasks)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-multiplatform-old",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "pull_request"
                     }
                 },
                 "spec": {
                     "pipelineSpec": {
                         "tasks": [
                             {
                                 "name": "build-amd64",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/amd64"}]
                             },
                             {
                                 "name": "build-s390x",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/s390x"}]
                             },
                             {
                                 "name": "build-ppc64le",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/ppc64le"}]
                             },
                             {
                                 "name": "build-arm64",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/arm64"}]
                             },
                             {
                                 "name": "build-darwin",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "darwin/amd64"}]
                             }
                         ]
                     },
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-pre-merge-build"
                 }
             }
         },
     
         "release_managed": {
             "name": "Release managed pipeline",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-release-managed",
                     "namespace": "default",
                     "labels": {
                         "appstudio.openshift.io/service": "release",
                         "pipelines.appstudio.openshift.io/type": "managed"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "release-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-release"
                 }
             }
         },
     
         "release_tenant": {
             "name": "Release tenant pipeline",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-release-tenant",
                     "namespace": "default",
                     "labels": {
                         "appstudio.openshift.io/service": "release",
                         "pipelines.appstudio.openshift.io/type": "tenant"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "release-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             }
             ,
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-tenant-release"
                 }
             }
         },
     
         "mintmaker": {
             "name": "Mintmaker pipeline",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-mintmaker",
                     "namespace": "mintmaker"
                 },
                 "spec": {
                     "pipelineRef": {"name": "mintmaker-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-dependency-update"
                 }
             }
         },
     
         "integration_test_push": {
             "name": "Integration test pipeline (push event)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-integration-push",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push",
                         "test.appstudio.openshift.io/scenario": "integration-test"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "integration-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-test"
                 }
             }
         },
     
         "integration_test_pr": {
             "name": "Integration test pipeline (pull_request event)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-integration-pr",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "pull_request",
                         "test.appstudio.openshift.io/scenario": "integration-test"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "integration-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-pre-merge-test"
                 }
             }
         },
     
         "default_priority": {
             "name": "Pipeline with default priority",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-default-priority",
                     "namespace": "default"
                 },
                 "spec": {
                     "pipelineRef": {"name": "default-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-default"
                 }
             }
         },
     
         "aws_platforms_only": {
             "name": "Pipeline with only AWS-based platforms",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-aws-platforms-only",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["linux/arm64", "darwin/amd64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "2"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
         },
     
         "mixed_platforms_excluded_included": {
             "name": "Pipeline with mixed excluded and included platforms for AWS IP",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-mixed-platforms",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
    -    },
    -
    -    "user-specific-priority": {
    -        "name": "Multi-platform pipeline with user-specific priority (new style)",
    -        "pipelinerun": {
    -            "apiVersion": "tekton.dev/v1",
    -            "kind": "PipelineRun",
    -            "metadata": {
    -                "name": "test-user-specific-priority",
    -                "namespace": "default",
    -                "labels": {
    -                    "pipelinesascode.tekton.dev/event-type": "push",
    -                    "kueue.x-k8s.io/priority-class": "konflux-user-specific"
    -                }
    -            },
    -            "spec": {
    -                "pipelineRef": {"name": "build-pipeline"},
    -                "params": [
    -                    {
    -                        "name": "build-platforms",
    -                        "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
    -                    }
    -                ],
    -                "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
    -            }
    -        },
    -        "expected": {
    -            "annotations": {
    -                "kueue.konflux-ci.dev/requests-linux-amd64": "1",
    -                "kueue.konflux-ci.dev/requests-linux-s390x": "1",
    -                "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
    -                "kueue.konflux-ci.dev/requests-linux-arm64": "1",
    -                "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
    -                "kueue.konflux-ci.dev/requests-aws-ip": "3"
    -            },
    -            "labels": {
    -                "kueue.x-k8s.io/queue-name": "pipelines-queue",
    -                "kueue.x-k8s.io/priority-class": "konflux-user-specific"
    -            }
    -        }
     }
     
     
    -# Configuration combinations that can be applied to any PipelineRun
    -CONFIG_COMBINATIONS: Dict[str, ConfigCombination] = {
    -    "development": {
    -        "name": "Development config",
    -        "config_file": "components/kueue/development/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/development/tekton-kueue/kustomization.yaml"
    -    },
    -    "staging": {
    -        "name": "Staging config",
    -        "config_file": "components/kueue/staging/base/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/staging/base/tekton-kueue/kustomization.yaml"
    -    },
    -    "production": {
    -        "name": "Production config",
    -        "config_file": "components/kueue/production/base/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
    -    },
    -    "production-kflux-ocp-p01": {
    -        "name": "Production config",
    -        "config_file": "components/kueue/production/kflux-ocp-p01/config.yaml",
    -        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
    -    }
    -}
    -
    -# Test combinations: which PipelineRuns to test with which configs
    -# This creates a cartesian product of PipelineRuns and configs
    -TEST_COMBINATIONS: Dict[str, TestCombination] = {
    -    # Test all PipelineRuns with development config (default)
    -    "multiplatform_new_dev": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "development"
    -    },
    -    "multiplatform_old_dev": {
    -        "pipelinerun_key": "multiplatform_old",
    -        "config_key": "development"
    -    },
    -    "release_managed_dev": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "development"
    -    },
    -    "release_tenant_dev": {
    -        "pipelinerun_key": "release_tenant",
    -        "config_key": "development"
    -    },
    -    "mintmaker_dev": {
    -        "pipelinerun_key": "mintmaker",
    -        "config_key": "development"
    -    },
    -    "integration_test_push_dev": {
    -        "pipelinerun_key": "integration_test_push",
    -        "config_key": "development"
    -    },
    -    "integration_test_pr_dev": {
    -        "pipelinerun_key": "integration_test_pr",
    -        "config_key": "development"
    -    },
    -    "default_priority_dev": {
    -        "pipelinerun_key": "default_priority",
    -        "config_key": "development"
    -    },
    -    "aws_platforms_only_dev": {
    -        "pipelinerun_key": "aws_platforms_only",
    -        "config_key": "development"
    -    },
    -    "mixed_platforms_excluded_included_dev": {
    -        "pipelinerun_key": "mixed_platforms_excluded_included",
    -        "config_key": "development"
    -    },
    -
    -    # Test key PipelineRuns with staging config
    -    "multiplatform_new_staging": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "staging"
    -    },
    -    "release_managed_staging": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "staging"
    -    },
    -    "integration_test_push_staging": {
    -        "pipelinerun_key": "integration_test_push",
    -        "config_key": "staging"
    -    },
    -
    -    # Test key PipelineRuns with production config
    -    "multiplatform_new_production": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "production"
    -    },
    -    "release_managed_production": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "production"
    -    },
    -
    -    # Example: Test the same PipelineRun with different configs to show reusability
    -    "user-specific-priority_and_mixed_platforms_production-kflux-ocp-p01": {
    -        "pipelinerun_key": "user-specific-priority",
    -        "config_key": "production-kflux-ocp-p01"
    -    }
    -}
    -
    -
     class TektonKueueMutationTest(unittest.TestCase):
         """Test suite for tekton-kueue CEL expression mutations."""
     
         @classmethod
         def setUpClass(cls):
            -"""Set up test class - check prerequisites and pre-process configs."""
            -cls.processed_configs = check_prerequisites(should_print=False)
            -cls.repo_root = Path(__file__).parent.parent
            -print("Prerequisites validated for all tests.")
            +"""Set up test class - check prerequisites."""
            +info = check_prerequisites(should_print=False)
            +cls.tekton_kueue_image = info.image
            +cls.config_file = info.config_file
            +print(f"Using tekton-kueue image: {cls.tekton_kueue_image}")
     
        -def run_mutation_test(self, test_combination: TestCombination) -> Dict[str, Any]:
            +def run_mutation_test(self, test_data: Dict) -> Dict:
             """Run a single mutation test and return results."""
            -        # Get pre-processed configuration
            -        config_key = test_combination["config_key"]
            -        test_config = self.processed_configs[config_key]
            -
            -        # Get the PipelineRun definition
            -        pipelinerun_key = test_combination["pipelinerun_key"]
            -        pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
            -        pipelinerun = pipelinerun_data["pipelinerun"]
            +        pipelinerun = test_data["pipelinerun"]
     
             with tempfile.TemporaryDirectory() as temp_dir:
                 # Write the config file
                 config_path = Path(temp_dir) / "config.yaml"
                 pipelinerun_path = Path(temp_dir) / "pipelinerun.yaml"
     
                -            # Copy the test-specific config file
                +            # Copy the config file
                 import shutil
                -            shutil.copy2(test_config.config_file, config_path)
                +            shutil.copy2(self.config_file, config_path)
     
                 # Write the PipelineRun
                 with open(pipelinerun_path, 'w') as f:
                     yaml.dump(pipelinerun, f, default_flow_style=False)
     
                 # Set proper permissions
                 os.chmod(config_path, 0o644)
                 os.chmod(pipelinerun_path, 0o644)
                 os.chmod(temp_dir, 0o755)
     
                -            # Run the mutation with test-specific image
                +            # Run the mutation
                 cmd = [
                     "podman", "run", "--rm",
                     "-v", f"{temp_dir}:/workspace:z",
                    -test_config.image,
                    +self.tekton_kueue_image,
                     "mutate",
                     "--pipelinerun-file", "/workspace/pipelinerun.yaml",
                     "--config-dir", "/workspace"
                 ]
     
                 result = subprocess.run(cmd, capture_output=True, text=True)
     
                 if result.returncode != 0:
                     self.fail(f"Mutation failed: {result.stderr}")
     
                 # Parse the mutated PipelineRun
                 try:
                     mutated = yaml.safe_load(result.stdout)
                 except yaml.YAMLError as e:
                     self.fail(f"Failed to parse mutated YAML: {e}")
     
                 return mutated
     
        -    def validate_mutation_result(self, test_key: str, test_combination: TestCombination) -> None:
            +    def validate_mutation_result(self, test_key: str, test_data: Dict):
                 """Helper method to validate mutation results."""
                 with self.subTest(test=test_key):
                    -            # Get pre-processed configuration for logging
                    -            config_key = test_combination["config_key"]
                    -            test_config = self.processed_configs[config_key]
                    -            print(f"Running test '{test_key}' with config: {test_config.config_file}, image: {test_config.image}")
                    -
                    -            mutated = self.run_mutation_test(test_combination)
                    -
                    -            # Get expected results from the PipelineRun definition
                    -            pipelinerun_key = test_combination["pipelinerun_key"]
                    -            pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
                    -            expected = pipelinerun_data["expected"]
                    +            mutated = self.run_mutation_test(test_data)
                    +            expected = test_data["expected"]
     
                    -            original_metadata = pipelinerun_data["pipelinerun"].get("metadata", {})
                    +            original_metadata = test_data["pipelinerun"].get("metadata", {})
                     original_annotations = original_metadata.get("annotations", {}) or {}
                     original_labels = original_metadata.get("labels", {}) or {}
     
                     # Check annotations (full equality vs original + expected)
                     annotations = mutated.get("metadata", {}).get("annotations", {})
                     expected_annotations = expected["annotations"]
                     expected_annotations_full = {**original_annotations, **expected_annotations}
                     self.assertDictEqual(
                         annotations,
                         expected_annotations_full,
                         f"Annotations mismatch; expected {expected_annotations_full}, got {annotations}"
                     )
     
                     # Check labels (full equality vs original + expected)
                     labels = mutated.get("metadata", {}).get("labels", {})
                     expected_labels = expected["labels"]
                     expected_labels_full = {**original_labels, **expected_labels}
                     self.assertDictEqual(
                         labels,
                         expected_labels_full,
                         f"Labels mismatch; expected {expected_labels_full}, got {labels}"
                     )
     
             def test_all_mutations(self):
                 """Test all tekton-kueue mutation scenarios."""
                -        for test_key, test_combination in TEST_COMBINATIONS.items():
                -            self.validate_mutation_result(test_key, test_combination)
                +        for test_key, test_data in TEST_PIPELINERUNS.items():
                +            self.validate_mutation_result(test_key, test_data)
     
     
     if __name__ == "__main__":
         import argparse
     
         parser = argparse.ArgumentParser(description="Test tekton-kueue CEL expressions")
         parser.add_argument("--check-setup", action="store_true",
                            help="Check if prerequisites are met and show configuration")
         parser.add_argument("--verbose", "-v", action="store_true",
                            help="Run tests with verbose output")
     
         # Parse known args to allow unittest args to pass through
         args, unknown = parser.parse_known_args()
     
         if args.check_setup:
             try:
                -            processed_configs = check_prerequisites(should_print=True)
                +            info = check_prerequisites(should_print=True)
             except Exception as e:
                 print(f"✗ {e}")
                 sys.exit(1)
     
             print("\n✅ All prerequisites met! Ready to run tests.")
             print("Run: python hack/test-tekton-kueue-config.py")
             print("\nNote: Tests will FAIL (not skip) if any prerequisites are missing.")
     
         else:
             # Run unittest with remaining args
    ```

5.  **Kueue `config.yaml` Deletion for `kflux-ocp-p01`**
    *   **Issue**: The `components/kueue/production/kflux-ocp-p01/config.yaml` file has been deleted, and its `configMapGenerator` removed from `kustomization.yaml`. This means `kflux-ocp-p01` will now implicitly use the `config.yaml` from `components/kueue/production/base/tekton-kueue/config.yaml`. While this might be intentional, it's a significant change in configuration for a specific production cluster.
    *   **File**: `components/kueue/production/kflux-ocp-p01/config.yaml` (deleted) and `components/kueue/production/kflux-ocp-p01/kustomization.yaml`
    *   **Suggestion**: Confirm that the `production/base` Kueue configuration is indeed the desired and fully tested configuration for `kflux-ocp-p01`. If there were specific requirements for `kflux-ocp-p01` that were handled by its dedicated `config.yaml`, ensure they are now covered by the base or are no longer needed.

### Minor Observations:

*   **Component Version Bumps**: Numerous components (`build-service`, `image-controller`, `integration`, `konflux-ui`, `mintmaker`, `monitoring/grafana/release`, `multi-platform-controller`, `project-controller`, `release`) have their image tags and Git references updated. These are standard updates, assuming the new versions have been tested.
*   **Kueue ClusterQueue Changes**: The `components/kueue/production/kflux-rhel-p01/queue-config/cluster-queue.yaml` has significant changes to `coveredResources` and `nominalQuota`, including the removal of `linux-d160-*` resources and a reduction in `linux-s390x` quota. These changes align with the `multi-platform-controller` host configuration updates and seem consistent.
*   **Multi-Platform Controller S390X Host Reduction**: The `components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml` shows a substantial reduction in S390X static hosts and the removal of `linux-large/s390x` platforms. This is a significant change in builder capacity for S390X. Ensure this reduction is intentional and has been communicated to relevant teams.

Please address the identified issues, especially the `environment` mismatches for Squid and Trust Manager, and the regression in Kueue test coverage.

Copy link
Contributor

github-actions bot commented Sep 8, 2025

Code Review by Gemini

The following issues and suggestions have been identified in the provided diff:

1. argo-cd-apps/base/member/infra-deployments/squid/squid.yaml and trust-manager/trust-manager.yaml - Hardcoded Environment in ApplicationSet

The ApplicationSet resources for squid and trust-manager hardcode the environment value within their clusters generator. This means that regardless of the actual cluster's environment (development or staging), squid will always attempt to deploy its development configuration, and trust-manager its staging configuration, to all clusters matching appstudio.redhat.com/member-cluster: "true". This contradicts the commit body's intent to deploy squid to "development and staging environments" and makes trust-manager deployment inflexible.

File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -10,11 +10,10 @@
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
-                environment: development
                 clusterDir: ""
           - list:
               elements: []
   template:
     metadata:
@@ -22,7 +21,7 @@
     spec:
       project: default
       source:
-        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
+        path: '{{values.sourceRoot}}/{{labels.appstudio.redhat.com/environment}}/{{values.clusterDir}}'
         repoURL: https://github.com/redhat-appstudio/infra-deployments.git
         targetRevision: main
       destination:

File: argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml

--- a/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
@@ -10,11 +10,10 @@
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/trust-manager
-                environment: staging
                 clusterDir: ""
           - list:
               elements: []
   template:
     metadata:
@@ -22,7 +21,7 @@
     spec:
       project: default
       source:
-        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
+        path: '{{values.sourceRoot}}/{{labels.appstudio.redhat.com/environment}}/{{values.clusterDir}}'
         repoURL: https://github.com/redhat-appstudio/infra-deployments.git
         targetRevision: main
       destination:

Suggestion:
Modify the ApplicationSet to dynamically determine the environment based on cluster labels. This assumes that member clusters have a label like appstudio.redhat.com/environment set to development or staging.

2. components/squid/development/squid-helm-generator.yaml and components/squid/staging/squid-helm-generator.yaml - Inconsistent Helm Chart Environment

The squid-helm-generator.yaml files for both development and staging environments specify environment: release in valuesInline. This value is passed directly to the Helm chart. It should ideally reflect the actual environment (development or staging) for consistency and correct Helm chart behavior.

File: components/squid/development/squid-helm-generator.yaml

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -10,7 +10,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:
       cpu: 100m

File: components/squid/staging/squid-helm-generator.yaml

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -10,7 +10,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:
       cpu: 100m

Suggestion:
Update the environment value in valuesInline to match the respective environment (development or staging).

3. hack/test-tekton-kueue-config.py - Test Coverage Regression

The hack/test-tekton-kueue-config.py script has been significantly refactored, removing test coverage for staging and production Kueue configurations. Previously, it tested against multiple configurations, including production-kflux-ocp-p01. While components/kueue/production/kflux-ocp-p01/config.yaml was deleted, the test script should still ensure that other environments (like staging and the generic production base) are covered, or that the removal of these tests is intentional and justified by a change in the configuration strategy.

File: hack/test-tekton-kueue-config.py

--- a/hack/test-tekton-kueue-config.py
+++ b/hack/test-tekton-kueue-config.py
@@ -1,101 +1,89 @@
 #!/usr/bin/env python3
 """
 Tekton-Kueue Configuration Test
 
 A comprehensive test suite that validates the CEL expressions in the tekton-kueue configuration by:
 
-1. **Reading configuration dynamically** from specified config files
-2. **Getting the image** from specified kustomization files
+1. **Reading configuration dynamically** from `components/kueue/development/tekton-kueue/config.yaml`
+2. **Getting the image** from `components/kueue/staging/base/tekton-kueue/kustomization.yaml`
 3. **Running mutations** using the actual tekton-kueue container via podman
 4. **Validating results** against expected annotations, labels, and priority classes
 
 Usage:
     # Check if all prerequisites are met
     python hack/test-tekton-kueue-config.py --check-setup
 
     # Run all tests
     python hack/test-tekton-kueue-config.py
 
     # Run tests with verbose output
     python hack/test-tekton-kueue-config.py --verbose
 
 Test Scenarios:
-    Each test can now specify its own config file and kustomization file,
-    allowing testing of multiple configurations and images.
+    The test covers all CEL expressions in the configuration:
+
+    1. **Multi-platform Resource Requests**:
+       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-*` annotations
+       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-*` annotations
+
+    2. **AWS IP Resource Requests**:
+       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-aws-ip` annotations
+         for platforms NOT in the excluded list (linux/ppc64le, linux/s390x, linux-x86-64, local, localhost, linux/amd64)
+       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-aws-ip` annotations
+         for platforms NOT in the excluded list
+
+    3. **Priority Assignment Logic**:
+       - Push events → `konflux-post-merge-build`
+       - Pull requests → `konflux-pre-merge-build`
+       - Integration test push → `konflux-post-merge-test`
+       - Integration test PR → `konflux-pre-merge-test`
+       - Release managed → `konflux-release`
+       - Release tenant → `konflux-tenant-release`
+       - Mintmaker namespace → `konflux-dependency-update`
+       - Default → `konflux-default`
+
+    4. **Queue Assignment**: All PipelineRuns get `kueue.x-k8s.io/queue-name: pipelines-queue`
 
 Prerequisites:
     - Python 3 with PyYAML
     - Podman (for running the tekton-kueue container)
-    - Access to the tekton-kueue images specified in the kustomizations
+    - Access to the tekton-kueue image specified in the kustomization
 
 CI/CD Integration:
     The test runs automatically on pull requests via the GitHub action
     `.github/workflows/test-tekton-kueue-config.yaml` when:
     - Changes are made to `components/kueue/**`
     - The test script itself is modified
     - The workflow file is modified
 
     The test will **FAIL** (not skip) if any prerequisites are missing, ensuring
     issues are caught early in CI/CD pipelines.
 """
 
 import subprocess
 import tempfile
 import os
 import yaml
 import unittest
 from pathlib import Path
-from typing import Dict, Any, TypedDict
+from typing import Dict
 from dataclasses import dataclass
 import sys
 
 
 @dataclass
-class TestConfig:
+class Prerequisites:
+    image: str
+    podman_version: str
     config_file: Path
     kustomization_file: Path
-    image: str
-
-
-class ConfigCombination(TypedDict):
-    config_file: str
-    kustomization_file: str
-
-
-class TestCombination(TypedDict):
-    pipelinerun_key: str
-    config_key: str
-
-
-class PipelineRunMetadata(TypedDict, total=False):
-    name: str
-    namespace: str
-    labels: Dict[str, str]
-    annotations: Dict[str, str]
-
-
-class PipelineRunDefinition(TypedDict):
-    apiVersion: str
-    kind: str
-    metadata: PipelineRunMetadata
-    spec: Dict[str, Any]  # More flexible since PipelineRun specs can vary
-
-
-class ExpectedResults(TypedDict):
-    annotations: Dict[str, str]
-    labels: Dict[str, str]
-
-
-class PipelineRunTestData(TypedDict):
-    pipelinerun: PipelineRunDefinition
-    expected: ExpectedResults
 
 
 def get_tekton_kueue_image(kustomization_file: Path) -> str:
     """Read the tekton-kueue image from the given kustomization file."""
     try:
         with open(kustomization_file, 'r') as f:
             kustomization = yaml.safe_load(f)
 
         # Look for the tekton-kueue image in the images section
         images = kustomization.get('images', [])
         for image in images:
             if image.get('name') == 'tekton-kueue':
                 new_name = image.get('newName', '')
                 new_tag = image.get('newTag', '')
                 if new_name and new_tag:
                     return f"{new_name}:{new_tag}"
 
         raise ValueError("tekton-kueue image not found in kustomization")
 
     except Exception as e:
         raise RuntimeError(f"Failed to read tekton-kueue image from {kustomization_file}: {e}")
 
-def resolve_path(path_str: str, repo_root: Path) -> Path:
-    """Resolve a path string to an absolute Path, handling both relative and absolute paths."""
-    if Path(path_str).is_absolute():
-        return Path(path_str)
-    return repo_root / path_str
-
-
-def validate_config_combination(config_key: str, repo_root: Path) -> TestConfig:
-    """Validate and resolve config and kustomization files for a config combination."""
-    config_data = CONFIG_COMBINATIONS[config_key]
-
-    config_file = resolve_path(config_data["config_file"], repo_root)
-    kustomization_file = resolve_path(config_data["kustomization_file"], repo_root)
-
-    # Validate files exist
-    if not config_file.exists():
-        raise FileNotFoundError(f"Config file not found for config '{config_key}': {config_file}")
-
-    if not kustomization_file.exists():
-        raise FileNotFoundError(f"Kustomization file not found for config '{config_key}': {kustomization_file}")
-
-    # Get image from kustomization
-    image = get_tekton_kueue_image(kustomization_file)
-
-    return TestConfig(
-        config_file=config_file,
-        kustomization_file=kustomization_file,
-        image=image
-    )
-
-
-def check_prerequisites(should_print: bool = True) -> Dict[str, TestConfig]:
-    """Check that all prerequisites are available and pre-process config combinations."""
-    messages = ["Checking prerequisites..."]
-    repo_root = Path(__file__).parent.parent
-
-    # Check podman availability
-    try:
-        result = subprocess.run(["podman", "--version"], capture_output=True, check=True, text=True)
-        podman_version = result.stdout.strip()
-        messages.append(f"✓ Podman available: {podman_version}")
-    except (subprocess.CalledProcessError, FileNotFoundError):
-        raise RuntimeError("Podman not available")
-
-    # Pre-process all unique config combinations
-    processed_configs: Dict[str, TestConfig] = {}
-
-    for _, test_combination in TEST_COMBINATIONS.items():
-        config_key = test_combination["config_key"]
-
-        # Only process each config combination once
-        if config_key not in processed_configs:
-            try:
-                config = validate_config_combination(config_key, repo_root)
-                processed_configs[config_key] = config
-                messages.append(f"✓ Config '{config_key}': {config.config_file}, image={config.image}")
-            except Exception as e:
-                raise RuntimeError(f"Config '{config_key}' validation failed: {e}")
+
+def check_prerequisites(should_print: bool = True) -> Prerequisites:
+    """Check that all prerequisites are available.
+
+    Returns a Prerequisites object with discovered info (image, podman_version)
+    on success. Raises an exception on failure.
+    """
+    messages = ["Checking prerequisites..."]
+
+    # Compute repo paths locally
+    repo_root = Path(__file__).parent.parent
+    config_file = repo_root / "components/kueue/development/tekton-kueue/config.yaml"
+    kustomization_file = repo_root / "components/kueue/development/tekton-kueue/kustomization.yaml"
+
+    # Config file
+    if not config_file.exists():
+        raise FileNotFoundError(f"Config file not found: {config_file}")
+    messages.append(f"\u2713 Config file found: {config_file}")
+
+    # Kustomization file
+    if not kustomization_file.exists():
+        raise FileNotFoundError(f"Kustomization file not found: {kustomization_file}")
+    messages.append(f"\u2713 Kustomization file found: {kustomization_file}")
+
+    # Image from kustomization
+    image = get_tekton_kueue_image(kustomization_file)
+    messages.append(f"\u2713 Tekton-kueue image: {image}")
+
+    # Podman availability
+    result = subprocess.run(["podman", "--version"], capture_output=True, check=True, text=True)
+    podman_version = result.stdout.strip()
+    messages.append(f"\u2713 Podman available: {podman_version}")
 
     if should_print:
-        for message in messages:
-            print(message)
+        for line in messages:
+            print(line)
 
-    return processed_configs
+    return Prerequisites(
+        image=image,
+        podman_version=podman_version,
+        config_file=config_file,
+        kustomization_file=kustomization_file,
+    )
 
-# Test PipelineRun definitions (reusable across different configs)
-PIPELINERUN_DEFINITIONS: Dict[str, PipelineRunTestData] = {
+# Test PipelineRun definitions
+TEST_PIPELINERUNS = {
     "multiplatform_new": {
         "name": "Multi-platform pipeline (new style with build-platforms parameter)",
         "pipelinerun": {
             "apiVersion": "tekton.dev/v1",
             "kind": "PipelineRun",
             "metadata": {
                 "name": "test-multiplatform-new",
                 "namespace": "default",
                 "labels": {
                     "pipelinesascode.tekton.dev/event-type": "push"
                 }
             },
             "spec": {
                 "pipelineRef": {"name": "build-pipeline"},
                 "params": [
                     {
                         "name": "build-platforms",
                         "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
                     }
                 ],
                 "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
             }
         },
         "expected": {
             "annotations": {
                 "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                 "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                 "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                 "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                 "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                 "kueue.konflux-ci.dev/requests-aws-ip": "3"
             },
             "labels": {
                 "kueue.x-k8s.io/queue-name": "pipelines-queue",
                 "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
             }
         }
-    },
-
-    "user-specific-priority": {
-        "name": "Multi-platform pipeline with user-specific priority (new style)",
-        "pipelinerun": {
-            "apiVersion": "tekton.dev/v1",
-            "kind": "PipelineRun",
-            "metadata": {
-                "name": "test-user-specific-priority",
-                "namespace": "default",
-                "labels": {
-                    "pipelinesascode.tekton.dev/event-type": "push",
-                    "kueue.x-k8s.io/priority-class": "konflux-user-specific"
-                }
-            },
-            "spec": {
-                "pipelineRef": {"name": "build-pipeline"},
-                "params": [
-                    {
-                        "name": "build-platforms",
-                        "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
-                    }
-                ],
-                "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
-            }
-        },
-        "expected": {
-            "annotations": {
-                "kueue.konflux-ci.dev/requests-linux-amd64": "1",
-                "kueue.konflux-ci.dev/requests-linux-s390x": "1",
-                "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
-                "kueue.konflux-ci.dev/requests-linux-arm64": "1",
-                "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
-                "kueue.konflux-ci.dev/requests-aws-ip": "3"
-            },
-            "labels": {
-                "kueue.x-k8s.io/queue-name": "pipelines-queue",
-                "kueue.x-k8s.io/priority-class": "konflux-user-specific"
-            }
-        }
-    },
-
-}
-
-# Configuration combinations that can be applied to any PipelineRun
-CONFIG_COMBINATIONS: Dict[str, ConfigCombination] = {
-    "development": {
-        "name": "Development config",
-        "config_file": "components/kueue/development/tekton-kueue/config.yaml",
-        "kustomization_file": "components/kueue/development/tekton-kueue/kustomization.yaml"
-    },
-    "staging": {
-        "name": "Staging config",
-        "config_file": "components/kueue/staging/base/tekton-kueue/config.yaml",
-        "kustomization_file": "components/kueue/staging/base/tekton-kueue/kustomization.yaml"
-    },
-    "production": {
-        "name": "Production config",
-        "config_file": "components/kueue/production/base/tekton-kueue/config.yaml",
-        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
-    },
-    "production-kflux-ocp-p01": {
-        "name": "Production config",
-        "config_file": "components/kueue/production/kflux-ocp-p01/config.yaml",
-        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
-    }
-}
-
-# Test combinations: which PipelineRuns to test with which configs
-# This creates a cartesian product of PipelineRuns and configs
-TEST_COMBINATIONS: Dict[str, TestCombination] = {
-    # Test all PipelineRuns with development config (default)
-    "multiplatform_new_dev": {
-        "pipelinerun_key": "multiplatform_new",
-        "config_key": "development"
-    },
-    "multiplatform_old_dev": {
-        "pipelinerun_key": "multiplatform_old",
-        "config_key": "development"
-    },
-    "release_managed_dev": {
-        "pipelinerun_key": "release_managed",
-        "config_key": "development"
-    },
-    "release_tenant_dev": {
-        "pipelinerun_key": "release_tenant",
-        "config_key": "development"
-    },
-    "mintmaker_dev": {
-        "pipelinerun_key": "mintmaker",
-        "config_key": "development"
-    },
-    "integration_test_push_dev": {
-        "pipelinerun_key": "integration_test_push",
-        "config_key": "development"
-    },
-    "integration_test_pr_dev": {
-        "pipelinerun_key": "integration_test_pr",
-        "config_key": "development"
-    },
-    "default_priority_dev": {
-        "pipelinerun_key": "default_priority",
-        "config_key": "development"
-    },
-    "aws_platforms_only_dev": {
-        "pipelinerun_key": "aws_platforms_only",
-        "config_key": "development"
-    },
-    "mixed_platforms_excluded_included_dev": {
-        "pipelinerun_key": "mixed_platforms_excluded_included",
-        "config_key": "development"
-    },
-
-    # Test key PipelineRuns with staging config
-    "multiplatform_new_staging": {
-        "pipelinerun_key": "multiplatform_new",
-        "config_key": "staging"
-    },
-    "release_managed_staging": {
-        "pipelinerun_key": "release_managed",
-        "config_key": "staging"
-    },
-    "integration_test_push_staging": {
-        "pipelinerun_key": "integration_test_push",
-        "config_key": "staging"
-    },
-
-    # Test key PipelineRuns with production config
-    "multiplatform_new_production": {
-        "pipelinerun_key": "multiplatform_new",
-        "config_key": "production"
-    },
-    "release_managed_production": {
-        "pipelinerun_key": "release_managed",
-        "config_key": "production"
-    },
-
-    # Example: Test the same PipelineRun with different configs to show reusability
-    "user-specific-priority_and_mixed_platforms_production-kflux-ocp-p01": {
-        "pipelinerun_key": "user-specific-priority",
-        "config_key": "production-kflux-ocp-p01"
     }
 }
 
 
 class TektonKueueMutationTest(unittest.TestCase):
     """Test suite for tekton-kueue CEL expression mutations."""
 
     @classmethod
     def setUpClass(cls):
-        """Set up test class - check prerequisites and pre-process configs."""
-        cls.processed_configs = check_prerequisites(should_print=False)
-        cls.repo_root = Path(__file__).parent.parent
-        print("Prerequisites validated for all tests.")
+        """Set up test class - check prerequisites."""
+        info = check_prerequisites(should_print=False)
+        cls.tekton_kueue_image = info.image
+        cls.config_file = info.config_file
+        print(f"Using tekton-kueue image: {cls.tekton_kueue_image}")
 
-    def run_mutation_test(self, test_combination: TestCombination) -> Dict[str, Any]:
+    def run_mutation_test(self, test_data: Dict) -> Dict:
         """Run a single mutation test and return results."""
-        # Get pre-processed configuration
-        config_key = test_combination["config_key"]
-        test_config = self.processed_configs[config_key]
-
-        # Get the PipelineRun definition
-        pipelinerun_key = test_combination["pipelinerun_key"]
-        pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
-        pipelinerun = pipelinerun_data["pipelinerun"]
+        pipelinerun = test_data["pipelinerun"]
 
         with tempfile.TemporaryDirectory() as temp_dir:
             # Write the config file
             config_path = Path(temp_dir) / "config.yaml"
             pipelinerun_path = Path(temp_dir) / "pipelinerun.yaml"
 
-            # Copy the test-specific config file
+            # Copy the config file
             import shutil
-            shutil.copy2(test_config.config_file, config_path)
+            shutil.copy2(self.config_file, config_path)
 
             # Write the PipelineRun
             with open(pipelinerun_path, 'w') as f:
                 yaml.dump(pipelinerun, f, default_flow_style=False)
 
             # Set proper permissions
             os.chmod(config_path, 0o644)
             os.chmod(pipelinerun_path, 0o644)
             os.chmod(temp_dir, 0o755)
 
-            # Run the mutation with test-specific image
+            # Run the mutation
             cmd = [
                 "podman", "run", "--rm",
                 "-v", f"{temp_dir}:/workspace:z",
-                test_config.image,
+                self.tekton_kueue_image,
                 "mutate",
                 "--pipelinerun-file", "/workspace/pipelinerun.yaml",
                 "--config-dir", "/workspace"
             ]
 
             result = subprocess.run(cmd, capture_output=True, text=True)
 
             if result.returncode != 0:
                 self.fail(f"Mutation failed: {result.stderr}")
 
             # Parse the mutated PipelineRun
             try:
                 mutated = yaml.safe_load(result.stdout)
             except yaml.YAMLError as e:
                 self.fail(f"Failed to parse mutated YAML: {e}")
 
             return mutated
 
-    def validate_mutation_result(self, test_key: str, test_combination: TestCombination) -> None:
+    def validate_mutation_result(self, test_key: str, test_data: Dict):
         """Helper method to validate mutation results."""
         with self.subTest(test=test_key):
-            # Get pre-processed configuration for logging
-            config_key = test_combination["config_key"]
-            test_config = self.processed_configs[config_key]
-            print(f"Running test '{test_key}' with config: {test_config.config_file}, image: {test_config.image}")
-
-            mutated = self.run_mutation_test(test_combination)
-
-            # Get expected results from the PipelineRun definition
-            pipelinerun_key = test_combination["pipelinerun_key"]
-            pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
-            expected = pipelinerun_data["expected"]
+            mutated = self.run_mutation_test(test_data)
+            expected = test_data["expected"]
 
-            original_metadata = pipelinerun_data["pipelinerun"].get("metadata", {})
+            original_metadata = test_data["pipelinerun"].get("metadata", {})
             original_annotations = original_metadata.get("annotations", {}) or {}
             original_labels = original_metadata.get("labels", {}) or {}
 
             # Check annotations (full equality vs original + expected)
             annotations = mutated.get("metadata", {}).get("annotations", {})
             expected_annotations = expected["annotations"]
             expected_annotations_full = {**original_annotations, **expected_annotations}
             self.assertDictEqual(
                 annotations,
                 expected_annotations_full,
                 f"Annotations mismatch; expected {expected_annotations_full}, got {annotations}"
             )
 
             # Check labels (full equality vs original + expected)
             labels = mutated.get("metadata", {}).get("labels", {})
             expected_labels = expected["labels"]
             expected_labels_full = {**original_labels, **expected_labels}
             self.assertDictEqual(
                 labels,
                 expected_labels_full,
                 f"Labels mismatch; expected {expected_labels_full}, got {labels}"
             )
 
     def test_all_mutations(self):
         """Test all tekton-kueue mutation scenarios."""
-        for test_key, test_combination in TEST_COMBINATIONS.items():
-            self.validate_mutation_result(test_key, test_combination)
+        for test_key, test_data in TEST_PIPELINERUNS.items():
+            self.validate_mutation_result(test_key, test_data)
 
 
 if __name__ == "__main__":
     import argparse
 
     parser = argparse.ArgumentParser(description="Test tekton-kueue CEL expressions")
     parser.add_argument("--check-setup", action="store_true",
                        help="Check if prerequisites are met and show configuration")
     parser.add_argument("--verbose", "-v", action="store_true",
                        help="Run tests with verbose output")
 
     # Parse known args to allow unittest args to pass through
     args, unknown = parser.parse_known_args()
 
     if args.check_setup:
         try:
-            processed_configs = check_prerequisites(should_print=True)
+            info = check_prerequisites(should_print=True)
         except Exception as e:
             print(f"✗ {e}")
             sys.exit(1)
 
         print("\n✅ All prerequisites met! Ready to run tests.")
         print("Run: python hack/test-tekton-kueue-config.py")
         print("\nNote: Tests will FAIL (not skip) if any prerequisites are missing.")
 
     else:
         # Run unittest with remaining args

Suggestion:
Reintroduce test cases for staging and production Kueue configurations. If specific config.yaml files for certain production clusters (like kflux-ocp-p01) are no longer needed, update the test suite to reflect the new, consolidated configuration approach. Ensure that all relevant environments have adequate test coverage.

4. components/konflux-info/production/*/banner-content.yaml - Future Maintenance Banner Date

Several banner-content.yaml files in production environments include a maintenance banner scheduled for "Thursday, September 4th at 13:00 UTC" in the year 2025. This date is quite far in the future and might be a typo (e.g., for 2024) or a placeholder that should not be committed so far in advance.

File: components/konflux-info/production/kflux-prd-rh03/banner-content.yaml (and similar files)

--- a/components/konflux-info/production/kflux-prd-rh03/banner-content.yaml
+++ b/components/konflux-info/production/kflux-prd-rh03/banner-content.yaml
@@ -1,2 +1,15 @@
-# Only the first banner will be displayed. Put the one to display at the top and remove any which are no longer relevant
-[]
+- type: danger
+  summary: "🚨 **Release pipelines are failing**: Release pipelines will fail due to an issue with image signing. The issue is being tracked in **[ITN-2025-00221](https://web-rca.devshift.net/incident/ITN-2025-00221)**."
+- type: warning
+  summary: "🚨 **ClamAV Task Issue**: There is a widespread issue in the clamav-task causing build failures with Rego parse errors. The issue is being tracked in **[KFLUXSPRT-4929](https://issues.redhat.com/browse/KFLUXSPRT-4929)**."
+- type: info
+  summary: ✨ Hope you're having a great day. If you need a little whimsy, check out [this wonderful thing](https://videos.learning.redhat.com/media/A+whole+new+build/1_v9l8cton) shared by the ART team.
+- type: info
+  summary: "This cluster will be undergoing maintenance on **Thursday, September 4th at 13:00 UTC**.  The cluster will remain usable during this mainenance period."
+- type: info
+  summary: "🚨 This cluster is currently undergoing maintenance until 15:00 UTC and will remain usable during this mainenance period. 🚨"
+  startTime: "13:00"
+  endTime: "15:00"
+  year: 2025
+  month: 9
+  dayOfMonth: 4

Suggestion:
Verify the year 2025 for the maintenance banner. If it's a typo, correct it. If it's a placeholder, consider removing it until the maintenance is imminent to avoid confusion.

5. components/konflux-info/OWNERS - Significant Reduction in Approvers/Reviewers

The OWNERS file for konflux-info has undergone a substantial reduction in the list of approvers and reviewers. While this might be an intentional consolidation of ownership, it's a high-impact change that should be explicitly acknowledged and justified.

File: components/konflux-info/OWNERS

--- a/components/konflux-info/OWNERS
+++ b/components/konflux-info/OWNERS
@@ -1,49 +1,15 @@
 # See the OWNERS docs: https://go.k8s.io/owners
 
 approvers:
-# infra-team
-- sadlerap
-- eisraeli
-- filariow
-- gbenhaim
-- hugares
-- manish-jangra
-- dperaza4dustbit
-# sre-team
-- dgregor
-- jdcasey
-# ui-team
-- abhinandan13jan
-- caugello
-- CryptoRodeo
-- JoaoPedroPP
-- Katka92
-- sahil143
 - testcara
+- sahil143
 - rrosatti
-# others
-- arewm
-
-reviewers:
-# infra-team
-- sadlerap
-- eisraeli
-- filariow
-- gbenhaim
-- hugares
-- manish-jangra
-- dperaza4dustbit
-# sre-team
-- dgregor
-- jdcasey
-# ui-team
-- abhinandan13jan
-- caugello
-- CryptoRodeo
 - JoaoPedroPP
 - Katka92
-- sahil143
+
+reviewers:
 - testcara
+- sahil143
 - rrosatti
-# others
-- arewm
+- JoaoPedroPP
+- Katka92

Suggestion:
Confirm that this significant change in ownership is intentional and has been communicated to the affected teams.

6. components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml and host-config.yaml - Builder Capacity Reduction

The changes in external-secrets.yaml and host-config.yaml for multi-platform-controller in kflux-rhel-p01 indicate a significant reduction in s390x static hosts (from 15 to 3), the consolidation of s390x SSH keys, and the removal of several linux-d160-* dynamic platforms. These are major operational changes affecting builder capacity and configuration.

File: components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml

--- a/components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml
+++ b/components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml
@@ -105,56 +105,10 @@
     - extract:
         key: production/platform/terraform/generated/kflux-rhel-p01/ibm-ppc64le-ssh-key-us-east
   refreshInterval: 1h
   secretStoreRef:
     kind: ClusterSecretStore
     name: appsre-stonesoup-vault
   target:
     creationPolicy: Owner
     deletionPolicy: Delete
     name: ibm-ppc64le-ssh-key
----
-apiVersion: external-secrets.io/v1beta1
-kind: ExternalSecret
-metadata:
-  name: ibm-s390x-ssh-key-regular
-  namespace: multi-platform-controller
-  labels:
-    build.appstudio.redhat.com/multi-platform-secret: "true"
-  annotations:
-    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
-    argocd.argoproj.io/sync-wave: "-1"
-spec:
-  dataFrom:
-    - extract:
-        key: production/platform/terraform/generated/kflux-rhel-p01/ibm-s390x-ssh-key-i6r3
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: appsre-stonesoup-vault
-  target:
-    creationPolicy: Owner
-    deletionPolicy: Delete
-    name: ibm-s390x-ssh-key-regular
----
-apiVersion: external-secrets.io/v1beta1
-kind: ExternalSecret
-metadata:
-  name: ibm-s390x-ssh-key-large-builder
-  namespace: multi-platform-controller
-  labels:
-    build.appstudio.redhat.com/multi-platform-secret: "true"
-  annotations:
-    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
-    argocd.argoproj.io/sync-wave: "-1"
-spec:
-  dataFrom:
-    - extract:
-        key: production/platform/terraform/generated/kflux-rhel-p01/ibm-s390x-ssh-key-wcrs
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: appsre-stonesoup-vault
-  target:
-    creationPolicy: Owner
-    deletionPolicy: Delete
-    name: ibm-s390x-ssh-key-large-builder

File: components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml

--- a/components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml
+++ b/components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml
@@ -9,38 +9,28 @@
   local-platforms: "\
     linux/x86_64,\
     local,\
     localhost,\
     "
   dynamic-platforms: "\
     linux/arm64,\
     linux/amd64,\
     linux-mlarge/arm64,\
     linux-mlarge/amd64,\
-    linux-d160-mlarge/arm64,\
-    linux-d160-mlarge/amd64,\
     linux-mxlarge/amd64,\
     linux-mxlarge/arm64,\
-    linux-d160-mxlarge/arm64,\
-    linux-d160-mxlarge/amd64,\
     linux-m2xlarge/amd64,\
     linux-m2xlarge/arm64,\
-    linux-d160-m2xlarge/arm64,\
-    linux-d160-m2xlarge/amd64,\
     linux-m4xlarge/amd64,\
     linux-m4xlarge/arm64,\
-    linux-d160-m4xlarge/arm64,\
-    linux-d160-m4xlarge/amd64,\
     linux-m8xlarge/amd64,\
     linux-m8xlarge/arm64,\
-    linux-d160-m8xlarge/arm64,\
-    linux-d160-m8xlarge/amd64,\
     linux-c6gd2xlarge/arm64,\
     linux-cxlarge/amd64,\
     linux-cxlarge/arm64,\
     linux-c2xlarge/amd64,\
     linux-c2xlarge/arm64,\
     linux-c4xlarge/amd64,\
     linux-c4xlarge/arm64,\
     linux-c8xlarge/amd64,\
     linux-c8xlarge/arm64,\
     linux-g6xlarge/amd64,\
@@ -78,133 +68,68 @@
   dynamic.linux-mlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-mlarge-arm64.instance-type: m6g.large
   dynamic.linux-mlarge-arm64.instance-tag: prod-arm64-mlarge
   dynamic.linux-mlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-mlarge-arm64.aws-secret: aws-account
   dynamic.linux-mlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-mlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-mlarge-arm64.max-instances: "250"
   dynamic.linux-mlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-mlarge-arm64.type: aws
-  dynamic.linux-d160-mlarge-arm64.region: us-east-1
-  dynamic.linux-d160-mlarge-arm64.ami: ami-03d6a5256a46c9feb
-  dynamic.linux-d160-mlarge-arm64.instance-type: m6g.large
-  dynamic.linux-d160-mlarge-arm64.instance-tag: prod-arm64-mlarge-d160
-  dynamic.linux-d160-mlarge-arm64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-mlarge-arm64.aws-secret: aws-account
-  dynamic.linux-d160-mlarge-arm64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-mlarge-arm64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-mlarge-arm64.max-instances: "250"
-  dynamic.linux-d160-mlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-mlarge-arm64.disk: "160"
-
   dynamic.linux-mxlarge-arm64.type: aws
   dynamic.linux-mxlarge-arm64.region: us-east-1
   dynamic.linux-mxlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-mxlarge-arm64.instance-type: m6g.xlarge
   dynamic.linux-mxlarge-arm64.instance-tag: prod-arm64-mxlarge
   dynamic.linux-mxlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-mxlarge-arm64.aws-secret: aws-account
   dynamic.linux-mxlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-mxlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-mxlarge-arm64.max-instances: "250"
   dynamic.linux-mxlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-mxlarge-arm64.type: aws
-  dynamic.linux-d160-mxlarge-arm64.region: us-east-1
-  dynamic.linux-d160-mxlarge-arm64.ami: ami-03d6a5256a46c9feb
-  dynamic.linux-d160-mxlarge-arm64.instance-type: m6g.xlarge
-  dynamic.linux-d160-mxlarge-arm64.instance-tag: prod-arm64-mxlarge-d160
-  dynamic.linux-d160-mxlarge-arm64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-mxlarge-arm64.aws-secret: aws-account
-  dynamic.linux-d160-mxlarge-arm64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-mxlarge-arm64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-mxlarge-arm64.max-instances: "250"
-  dynamic.linux-d160-mxlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-mxlarge-arm64.disk: "160"
-
   dynamic.linux-m2xlarge-arm64.type: aws
   dynamic.linux-m2xlarge-arm64.region: us-east-1
   dynamic.linux-m2xlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-m2xlarge-arm64.instance-type: m6g.2xlarge
   dynamic.linux-m2xlarge-arm64.instance-tag: prod-arm64-m2xlarge
   dynamic.linux-m2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m2xlarge-arm64.aws-secret: aws-account
   dynamic.linux-m2xlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-m2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m2xlarge-arm64.max-instances: "250"
   dynamic.linux-m2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m2xlarge-arm64.type: aws
-  dynamic.linux-d160-m2xlarge-arm64.region: us-east-1
-  dynamic.linux-d160-m2xlarge-arm64.ami: ami-03d6a5256a46c9feb
-  dynamic.linux-d160-m2xlarge-arm64.instance-type: m6g.2xlarge
-  dynamic.linux-d160-m2xlarge-arm64.instance-tag: prod-arm64-m2xlarge-d160
-  dynamic.linux-d160-m2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m2xlarge-arm64.aws-secret: aws-account
-  dynamic.linux-d160-m2xlarge-arm64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m2xlarge-arm64.max-instances: "250"
-  dynamic.linux-d160-m2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m2xlarge-arm64.disk: "160"
-
   dynamic.linux-m4xlarge-arm64.type: aws
   dynamic.linux-m4xlarge-arm64.region: us-east-1
   dynamic.linux-m4xlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-m4xlarge-arm64.instance-type: m6g.4xlarge
   dynamic.linux-m4xlarge-arm64.instance-tag: prod-arm64-m4xlarge
   dynamic.linux-m4xlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m4xlarge-arm64.aws-secret: aws-account
   dynamic.linux-m4xlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-m4xlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m4xlarge-arm64.max-instances: "250"
   dynamic.linux-m4xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m4xlarge-arm64.type: aws
-  dynamic.linux-d160-m4xlarge-arm64.region: us-east-1
-  dynamic.linux-d160-m4xlarge-arm64.ami: ami-03d6a5256a46c9feb
-  dynamic.linux-d160-m4xlarge-arm64.instance-type: m6g.4xlarge
-  dynamic.linux-d160-m4xlarge-arm64.instance-tag: prod-arm64-m4xlarge-d160
-  dynamic.linux-d160-m4xlarge-arm64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m4xlarge-arm64.aws-secret: aws-account
-  dynamic.linux-d160-m4xlarge-arm64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m4xlarge-arm64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m4xlarge-arm64.max-instances: "250"
-  dynamic.linux-d160-m4xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m4xlarge-arm64.disk: "160"
-
   dynamic.linux-m8xlarge-arm64.type: aws
   dynamic.linux-m8xlarge-arm64.region: us-east-1
   dynamic.linux-m8xlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-m8xlarge-arm64.instance-type: m6g.8xlarge
   dynamic.linux-m8xlarge-arm64.instance-tag: prod-arm64-m8xlarge
   dynamic.linux-m8xlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m8xlarge-arm64.aws-secret: aws-account
   dynamic.linux-m8xlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-m8xlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m8xlarge-arm64.max-instances: "250"
   dynamic.linux-m8xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m8xlarge-arm64.type: aws
-  dynamic.linux-d160-m8xlarge-arm64.region: us-east-1
-  dynamic.linux-d160-m8xlarge-arm64.ami: ami-03d6a5256a46c9feb
-  dynamic.linux-d160-m8xlarge-arm64.instance-type: m6g.8xlarge
-  dynamic.linux-d160-m8xlarge-arm64.instance-tag: prod-arm64-m8xlarge-d160
-  dynamic.linux-d160-m8xlarge-arm64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m8xlarge-arm64.aws-secret: aws-account
-  dynamic.linux-d160-m8xlarge-arm64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m8xlarge-arm64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m8xlarge-arm64.max-instances: "250"
-  dynamic.linux-d160-m8xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m8xlarge-arm64.disk: "160"
-
   dynamic.linux-c6gd2xlarge-arm64.type: aws
   dynamic.linux-c6gd2xlarge-arm64.region: us-east-1
   dynamic.linux-c6gd2xlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-c6gd2xlarge-arm64.instance-type: c6gd.2xlarge
   dynamic.linux-c6gd2xlarge-arm64.instance-tag: prod-arm64-c6gd2xlarge
   dynamic.linux-c6gd2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-c6gd2xlarge-arm64.aws-secret: aws-account
   dynamic.linux-c6gd2xlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-c6gd2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-c6gd2xlarge-arm64.max-instances: "250"
   dynamic.linux-c6gd2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
@@ -291,133 +216,68 @@
 
   dynamic.linux-mlarge-amd64.ami: ami-026ebd4cfe2c043b2
   dynamic.linux-mlarge-amd64.instance-type: m7a.large
   dynamic.linux-mlarge-amd64.instance-tag: prod-amd64-mlarge
   dynamic.linux-mlarge-amd64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-mlarge-amd64.aws-secret: aws-account
   dynamic.linux-mlarge-amd64.ssh-secret: aws-ssh-key
   dynamic.linux-mlarge-amd64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-mlarge-amd64.max-instances: "250"
   dynamic.linux-mlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-mlarge-amd64.type: aws
-  dynamic.linux-d160-mlarge-amd64.region: us-east-1
-  dynamic.linux-d160-mlarge-amd64.ami: ami-026ebd4cfe2c043b2
-  dynamic.linux-d160-mlarge-amd64.instance-type: m7a.large
-  dynamic.linux-d160-mlarge-amd64.instance-tag: prod-amd64-mlarge-d160
-  dynamic.linux-d160-mlarge-amd64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-mlarge-amd64.aws-secret: aws-account
-  dynamic.linux-d160-mlarge-amd64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-mlarge-amd64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-mlarge-amd64.max-instances: "250"
-  dynamic.linux-d160-mlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-mlarge-amd64.disk: "160"
-
   dynamic.linux-mxlarge-amd64.type: aws
   dynamic.linux-mxlarge-amd64.region: us-east-1
   dynamic.linux-mxlarge-amd64.ami: ami-026ebd4cfe2c043b2
   dynamic.linux-mxlarge-amd64.instance-type: m7a.xlarge
   dynamic.linux-mxlarge-amd64.instance-tag: prod-amd64-mxlarge
   dynamic.linux-mxlarge-amd64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-mxlarge-amd64.aws-secret: aws-account
   dynamic.linux-mxlarge-amd64.ssh-secret: aws-ssh-key
   dynamic.linux-mxlarge-amd64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-mxlarge-amd64.max-instances: "250"
   dynamic.linux-mxlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-mxlarge-amd64.type: aws
-  dynamic.linux-d160-mxlarge-amd64.region: us-east-1
-  dynamic.linux-d160-mxlarge-amd64.ami: ami-026ebd4cfe2c043b2
-  dynamic.linux-d160-mxlarge-amd64.instance-type: m7a.xlarge
-  dynamic.linux-d160-mxlarge-amd64.instance-tag: prod-amd64-mxlarge-d160
-  dynamic.linux-d160-mxlarge-amd64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-mxlarge-amd64.aws-secret: aws-account
-  dynamic.linux-d160-mxlarge-amd64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-mxlarge-amd64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-mxlarge-amd64.max-instances: "250"
-  dynamic.linux-d160-mxlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-mxlarge-amd64.disk: "160"
-
   dynamic.linux-m2xlarge-amd64.type: aws
   dynamic.linux-m2xlarge-amd64.region: us-east-1
   dynamic.linux-m2xlarge-amd64.ami: ami-026ebd4cfe2c043b2
   dynamic.linux-m2xlarge-amd64.instance-type: m7a.2xlarge
   dynamic.linux-m2xlarge-amd64.instance-tag: prod-amd64-m2xlarge
   dynamic.linux-m2xlarge-amd64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m2xlarge-amd64.aws-secret: aws-account
   dynamic.linux-m2xlarge-amd64.ssh-secret: aws-ssh-key
   dynamic.linux-m2xlarge-amd64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m2xlarge-amd64.max-instances: "250"
   dynamic.linux-m2xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m2xlarge-amd64.type: aws
-  dynamic.linux-d160-m2xlarge-amd64.region: us-east-1
-  dynamic.linux-d160-m2xlarge-amd64.ami: ami-026ebd4cfe2c043b2
-  dynamic.linux-d160-m2xlarge-amd64.instance-type: m7a.2xlarge
-  dynamic.linux-d160-m2xlarge-amd64.instance-tag: prod-amd64-m2xlarge-d160
-  dynamic.linux-d160-m2xlarge-amd64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m2xlarge-amd64.aws-secret: aws-account
-  dynamic.linux-d160-m2xlarge-amd64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m2xlarge-amd64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m2xlarge-amd64.max-instances: "250"
-  dynamic.linux-d160-m2xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m2xlarge-amd64.disk: "160"
-
   dynamic.linux-m4xlarge-amd64.type: aws
   dynamic.linux-m4xlarge-amd64.region: us-east-1
   dynamic.linux-m4xlarge-amd64.ami: ami-026ebd4cfe2c043b2
   dynamic.linux-m4xlarge-amd64.instance-type: m7a.4xlarge
   dynamic.linux-m4xlarge-amd64.instance-tag: prod-amd64-m4xlarge
   dynamic.linux-m4xlarge-amd64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m4xlarge-amd64.aws-secret: aws-account
   dynamic.linux-m4xlarge-amd64.ssh-secret: aws-ssh-key
   dynamic.linux-m4xlarge-amd64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m4xlarge-amd64.max-instances: "250"
   dynamic.linux-m4xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m4xlarge-amd64.type: aws
-  dynamic.linux-d160-m4xlarge-amd64.region: us-east-1
-  dynamic.linux-d160-m4xlarge-amd64.ami: ami-026ebd4cfe2c043b2
-  dynamic.linux-d160-m4xlarge-amd64.instance-type: m7a.4xlarge
-  dynamic.linux-d160-m4xlarge-amd64.instance-tag: prod-amd64-m4xlarge-d160
-  dynamic.linux-d160-m4xlarge-amd64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m4xlarge-amd64.aws-secret: aws-account
-  dynamic.linux-d160-m4xlarge-amd64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m4xlarge-amd64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m4xlarge-amd64.max-instances: "250"
-  dynamic.linux-d160-m4xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m4xlarge-amd64.disk: "160"
-
   dynamic.linux-m8xlarge-amd64.type: aws
   dynamic.linux-m8xlarge-amd64.region: us-east-1
   dynamic.linux-m8xlarge-amd64.ami: ami-026ebd4cfe2c043b2
   dynamic.linux-m8xlarge-amd64.instance-type: m7a.8xlarge
   dynamic.linux-m8xlarge-amd64.instance-tag: prod-amd64-m8xlarge
   dynamic.linux-m8xlarge-amd64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-m8xlarge-amd64.aws-secret: aws-account
   dynamic.linux-m8xlarge-amd64.ssh-secret: aws-ssh-key
   dynamic.linux-m8xlarge-amd64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-m8xlarge-amd64.max-instances: "250"
   dynamic.linux-m8xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
 
-  dynamic.linux-d160-m8xlarge-amd64.type: aws
-  dynamic.linux-d160-m8xlarge-amd64.region: us-east-1
-  dynamic.linux-d160-m8xlarge-amd64.ami: ami-026ebd4cfe2c043b2
-  dynamic.linux-d160-m8xlarge-amd64.instance-type: m7a.8xlarge
-  dynamic.linux-d160-m8xlarge-amd64.instance-tag: prod-amd64-m8xlarge-d160
-  dynamic.linux-d160-m8xlarge-amd64.key-name: kflux-rhel-p01-key-pair
-  dynamic.linux-d160-m8xlarge-amd64.aws-secret: aws-account
-  dynamic.linux-d160-m8xlarge-amd64.ssh-secret: aws-ssh-key
-  dynamic.linux-d160-m8xlarge-amd64.security-group-id: sg-0c67a834068be63d6
-  dynamic.linux-d160-m8xlarge-amd64.max-instances: "250"
-  dynamic.linux-d160-m8xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
-  dynamic.linux-d160-m8xlarge-amd64.disk: "160"
-
   # cpu:memory (1:2)
   dynamic.linux-cxlarge-arm64.type: aws
   dynamic.linux-cxlarge-arm64.region: us-east-1
   dynamic.linux-cxlarge-arm64.ami: ami-03d6a5256a46c9feb
   dynamic.linux-cxlarge-arm64.instance-type: c6g.xlarge
   dynamic.linux-cxlarge-arm64.instance-tag: prod-arm64-cxlarge
   dynamic.linux-cxlarge-arm64.key-name: kflux-rhel-p01-key-pair
   dynamic.linux-cxlarge-arm64.aws-secret: aws-account
   dynamic.linux-cxlarge-arm64.ssh-secret: aws-ssh-key
   dynamic.linux-cxlarge-arm64.security-group-id: sg-0c67a834068be63d6
   dynamic.linux-cxlarge-arm64.max-instances: "250"
@@ -626,129 +486,38 @@
 
     sed -n 's,.*\(ssh-.*\s\),\1,p' /root/.ssh/authorized_keys > /home/ec2-user/.ssh/authorized_keys
     chown ec2-user /home/ec2-user/.ssh/authorized_keys
     chmod 600 /home/ec2-user/.ssh/authorized_keys
     chmod 700 /home/ec2-user/.ssh
     restorecon -r /home/ec2-user
 
     --//--
 
   # S390X 16vCPU / 64GiB RAM / 1TB disk
-  host.s390x-static-0.address: "10.130.130.9"
-  host.s390x-static-0.platform: "linux/s390x"
-  host.s390x-static-0.user: "root"
-  host.s390x-static-0.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-0.concurrency: "4"
-
-  host.s390x-static-1.address: "10.130.130.21"
+  host.s390x-static-1.address: "10.130.79.109"
   host.s390x-static-1.platform: "linux/s390x"
   host.s390x-static-1.user: "root"
-  host.s390x-static-1.secret: "ibm-s390x-ssh-key-regular"
+  host.s390x-static-1.secret: "ibm-s390x-ssh-key"
   host.s390x-static-1.concurrency: "4"
 
-  host.s390x-static-2.address: "10.130.130.39"
+  host.s390x-static-2.address: "10.130.79.106"
   host.s390x-static-2.platform: "linux/s390x"
   host.s390x-static-2.user: "root"
-  host.s390x-static-2.secret: "ibm-s390x-ssh-key-regular"
+  host.s390x-static-2.secret: "ibm-s390x-ssh-key"
   host.s390x-static-2.concurrency: "4"
 
-  host.s390x-static-3.address: "10.130.130.6"
+  host.s390x-static-3.address: "10.130.79.137"
   host.s390x-static-3.platform: "linux/s390x"
   host.s390x-static-3.user: "root"
-  host.s390x-static-3.secret: "ibm-s390x-ssh-key-regular"
+  host.s390x-static-3.secret: "ibm-s390x-ssh-key"
   host.s390x-static-3.concurrency: "4"
 
-  host.s390x-static-4.address: "10.130.130.23"
-  host.s390x-static-4.platform: "linux/s390x"
-  host.s390x-static-4.user: "root"
-  host.s390x-static-4.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-4.concurrency: "4"
-
-  host.s390x-static-5.address: "10.130.130.38"
-  host.s390x-static-5.platform: "linux/s390x"
-  host.s390x-static-5.user: "root"
-  host.s390x-static-5.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-5.concurrency: "4"
-
-  host.s390x-static-6.address: "10.130.130.8"
-  host.s390x-static-6.platform: "linux/s390x"
-  host.s390x-static-6.user: "root"
-  host.s390x-static-6.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-6.concurrency: "4"
-
-  host.s390x-static-7.address: "10.130.130.22"
-  host.s390x-static-7.platform: "linux/s390x"
-  host.s390x-static-7.user: "root"
-  host.s390x-static-7.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-7.concurrency: "4"
-
-  host.s390x-static-8.address: "10.130.130.37"
-  host.s390x-static-8.platform: "linux/s390x"
-  host.s390x-static-8.user: "root"
-  host.s390x-static-8.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-8.concurrency: "4"
-
-  host.s390x-static-9.address: "10.130.130.7"
-  host.s390x-static-9.platform: "linux/s390x"
-  host.s390x-static-9.user: "root"
-  host.s390x-static-9.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-9.concurrency: "4"
-
-  host.s390x-static-10.address: "10.130.130.24"
-  host.s390x-static-10.platform: "linux/s390x"
-  host.s390x-static-10.user: "root"
-  host.s390x-static-10.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-10.concurrency: "4"
-
-  host.s390x-static-11.address: "10.130.130.40"
-  host.s390x-static-11.platform: "linux/s390x"
-  host.s390x-static-11.user: "root"
-  host.s390x-static-11.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-11.concurrency: "4"
-
-  host.s390x-static-12.address: "10.130.130.10"
-  host.s390x-static-12.platform: "linux/s390x"
-  host.s390x-static-12.user: "root"
-  host.s390x-static-12.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-12.concurrency: "4"
-
-  host.s390x-static-13.address: "10.130.130.25"
-  host.s390x-static-13.platform: "linux/s390x"
-  host.s390x-static-13.user: "root"
-  host.s390x-static-13.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-13.concurrency: "4"
-
-  host.s390x-static-14.address: "10.130.130.41"
-  host.s390x-static-14.platform: "linux/s390x"
-  host.s390x-static-14.user: "root"
-  host.s390x-static-14.secret: "ibm-s390x-ssh-key-regular"
-  host.s390x-static-14.concurrency: "4"
-
-  # S390X 32vCPU / 128GiB RAM / 1TB disk
-  host.s390x-large-static-0.address: "10.130.130.5"
-  host.s390x-large-static-0.platform: "linux-large/s390x"
-  host.s390x-large-static-0.user: "root"
-  host.s390x-large-static-0.secret: "ibm-s390x-ssh-key-large-builder"
-  host.s390x-large-static-0.concurrency: "4"
-
-  host.s390x-large-static-1.address: "10.130.130.20"
-  host.s390x-large-static-1.platform: "linux-large/s390x"
-  host.s390x-large-static-1.user: "root"
-  host.s390x-large-static-1.secret: "ibm-s390x-ssh-key-large-builder"
-  host.s390x-large-static-1.concurrency: "4"
-
-  host.s390x-large-static-2.address: "10.130.130.36"
-  host.s390x-large-static-2.platform: "linux-large/s390x"
-  host.s390x-large-static-2.user: "root"
-  host.s390x-large-static-2.secret: "ibm-s390x-ssh-key-large-builder"
-  host.s390x-large-static-2.concurrency: "4"
-
   # PPC64LE 4cores(32vCPU) / 128GiB RAM / 2TB disk
   host.ppc64le-static-0.address: "10.130.78.70"
   host.ppc64le-static-0.platform: "linux/ppc64le"
   host.ppc64le-static-0.user: "root"
   host.ppc64le-static-0.secret: "ibm-ppc64le-ssh-key"
   host.ppc64le-static-0.concurrency: "8"
 
   host.ppc64le-static-1.address: "10.130.78.73"
   host.ppc64le-static-1.platform: "linux/ppc64le"
   host.ppc64le-static-1.user: "root"

Suggestion:
Confirm with the multi-platform builder team that these changes in builder capacity and configuration are intentional and correctly reflect the current infrastructure and requirements. This also applies to the corresponding changes in components/kueue/production/kflux-rhel-p01/queue-config/cluster-queue.yaml.

Copy link
Contributor

github-actions bot commented Sep 9, 2025

Code Review by Gemini

The pull request introduces several changes, many of which appear unrelated to the stated goal of deploying Squid to development and staging environments. This makes the review process difficult and increases the risk of unintended side effects.

Here's a breakdown of the issues and suggested improvements:

1. Scope Creep / Unrelated Changes
The pull request includes a large number of changes that are not directly related to deploying Squid or Trust Manager. These include:

  • Updates to various component image tags (build-service, image-controller, integration, knative-eventing, konflux-ui, mintmaker, multi-platform-controller, release-service).
  • Changes to Velero backup schedules.
  • Modifications to kubelet.yaml regarding ignoreDifferences.
  • Updates to konflux-info/OWNERS and banner content.
  • Significant changes to Kueue configurations and resource flavors.
  • Changes to Prometheus monitoring metrics and relabel configurations.
  • Major modifications to multi-platform controller host configurations and external secrets.
  • Changes to Kyverno policies for konflux-rbac.
  • Removal of gosmee version patches.
  • Regression in Kueue test coverage.

Recommendation: These unrelated changes should be split into separate pull requests to allow for focused review and reduce the risk of introducing regressions.

2. Squid ApplicationSet Configuration Issues

  • Hardcoded Environment: The squid.yaml ApplicationSet hardcodes the environment to development. The commit message states deployment to "development and staging". This will only deploy to development clusters.
    • File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    --- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    @@ -13,7 +13,7 @@
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
    -            environment: development
    +            environment: '{{nameNormalized}}' # Or use a matrix generator for dev/staging
                 clusterDir: ""
           - list:
               elements: []
  • Hardcoded Helm Generator Environment: The squid-helm-generator.yaml also hardcodes the environment to release. This should be dynamic based on the target environment (development/staging).
    • File: components/squid/development/squid-helm-generator.yaml
    --- a/components/squid/development/squid-helm-generator.yaml
    +++ b/components/squid/development/squid-helm-generator.yaml
    @@ -13,7 +13,7 @@
     +    enabled: false
     +  cert-manager:
     +    enabled: false
    -  environment: release
    +  environment: development
     +  resources:
     +    requests:
     +      cpu: 100m
    • File: components/squid/staging/squid-helm-generator.yaml
    --- a/components/squid/staging/squid-helm-generator.yaml
    +++ b/components/squid/staging/squid-helm-generator.yaml
    @@ -13,7 +13,7 @@
     +    enabled: false
     +  cert-manager:
     +    enabled: false
    -  environment: release
    +  environment: staging
     +  resources:
     +    requests:
     +      cpu: 100m
  • selfHeal: false: For an infrastructure component like Squid, selfHeal: true is generally preferred to ensure the desired state is maintained automatically. If false is intentional, please provide justification.
    • File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    --- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
    @@ -29,7 +29,7 @@
       destination:
         namespace: proxy
         server: '{{server}}'
       syncPolicy:
         automated:
           prune: true
    -      selfHeal: false
    +      selfHeal: true # Consider changing to true for infrastructure components
         syncOptions:
           - CreateNamespace=true
         retry:

3. Trust Manager ApplicationSet Configuration Issues

  • Hardcoded Environment: The trust-manager.yaml ApplicationSet hardcodes the environment to staging. Similar to Squid, this will only deploy to staging clusters.
    • File: argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    --- a/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    @@ -13,7 +13,7 @@
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/trust-manager
    -            environment: staging
    +            environment: '{{nameNormalized}}' # Or use a matrix generator for dev/staging
                 clusterDir: ""
           - list:
               elements: []
  • Inconsistent Resource Requests/Limits: The resource requests and limits for Trust Manager are different between the development and staging Helm generators.
    • Development: cpu: 25m, memory: 64Mi
    • Staging: cpu: 100m, memory: 128Mi
      This inconsistency should be justified or standardized.
    • File: components/trust-manager/development/trust-manager-helm-generator.yaml
    --- a/components/trust-manager/development/trust-manager-helm-generator.yaml
    +++ b/components/trust-manager/development/trust-manager-helm-generator.yaml
    @@ -11,15 +11,15 @@
     releaseName: trust-manager
     valuesInline:
       resources:
         requests:
    -      cpu: 25m
    -      memory: 64Mi
    +      cpu: 100m # Consider aligning with staging or justifying difference
    +      memory: 128Mi # Consider aligning with staging or justifying difference
         limits:
    -      cpu: 25m
    -      memory: 64Mi
    +      cpu: 100m # Consider aligning with staging or justifying difference
    +      memory: 128Mi # Consider aligning with staging or justifying difference
       defaultPackage:
         resources:
           requests:
    -        cpu: 25m
    -        memory: 64Mi
    +        cpu: 100m # Consider aligning with staging or justifying difference
    +        memory: 128Mi # Consider aligning with staging or justifying difference
           limits:
    -        cpu: 25m
    -        memory: 64Mi
    +        cpu: 100m # Consider aligning with staging or justifying difference
    +        memory: 128Mi # Consider aligning with staging or justifying difference
  • selfHeal: false: Similar to Squid, selfHeal: true is generally preferred for infrastructure components. If false is intentional, please provide justification.
    • File: argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    --- a/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    +++ b/argo-cd-apps/base/member/infra-deployments/trust-manager/trust-manager.yaml
    @@ -29,7 +29,7 @@
       destination:
         namespace: cert-manager
         server: '{{server}}'
       syncPolicy:
         automated:
           prune: true
    -      selfHeal: false
    +      selfHeal: true # Consider changing to true for infrastructure components
         syncOptions:
           - CreateNamespace=true
         retry:

4. Kueue Test Coverage Regression

  • The changes to hack/test-tekton-kueue-config.py remove the ability to test Kueue configurations for staging and production environments, hardcoding it to only test the development configuration. This significantly reduces test coverage and should be reverted or fixed to ensure all relevant configurations are tested.
    • File: hack/test-tekton-kueue-config.py
    --- a/hack/test-tekton-kueue-config.py
    +++ b/hack/test-tekton-kueue-config.py
    @@ -1,101 +1,89 @@
     #!/usr/bin/env python3
     """
     Tekton-Kueue Configuration Test
     
     A comprehensive test suite that validates the CEL expressions in the tekton-kueue configuration by:
     
    -1. **Reading configuration dynamically** from specified config files
    -2. **Getting the image** from specified kustomization files
    +1. **Reading configuration dynamically** from `components/kueue/development/tekton-kueue/config.yaml`
    +2. **Getting the image** from `components/kueue/staging/base/tekton-kueue/kustomization.yaml`
     3. **Running mutations** using the actual tekton-kueue container via podman
     4. **Validating results** against expected annotations, labels, and priority classes
     
     Usage:
         # Check if all prerequisites are met
         python hack/test-tekton-kueue-config.py --check-setup
     
         # Run all tests
         python hack/test-tekton-kueue-config.py
     
         # Run tests with verbose output
         python hack/test-tekton-kueue-config.py --verbose
     
     Test Scenarios:
    -    Each test can now specify its own config file and kustomization file,
    -    allowing testing of multiple configurations and images.
    +    The test covers all CEL expressions in the configuration:
    +
    +    1. **Multi-platform Resource Requests**:
    +       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-*` annotations
    +       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-*` annotations
    +
    +    2. **AWS IP Resource Requests**:
    +       - New style: `build-platforms` parameter → `kueue.konflux-ci.dev/requests-aws-ip` annotations
    +         for platforms NOT in the excluded list (linux/ppc64le, linux/s390x, linux-x86-64, local, localhost, linux/amd64)
    +       - Old style: `PLATFORM` parameters in tasks → `kueue.konflux-ci.dev/requests-aws-ip` annotations
    +         for platforms NOT in the excluded list
    +
    +    3. **Priority Assignment Logic**:
    +       - Push events → `konflux-post-merge-build`
    +       - Pull requests → `konflux-pre-merge-build`
    +       - Integration test push → `konflux-post-merge-test`
    +       - Integration test PR → `konflux-pre-merge-test`
    +       - Release managed → `konflux-release`
    +       - Release tenant → `konflux-tenant-release`
    +       - Mintmaker namespace → `konflux-dependency-update`
    +       - Default → `konflux-default`
    +
    +    4. **Queue Assignment**: All PipelineRuns get `kueue.x-k8s.io/queue-name: pipelines-queue`
     
     Prerequisites:
         - Python 3 with PyYAML
         - Podman (for running the tekton-kueue container)
    -    - Access to the tekton-kueue images specified in the kustomizations
    +    - Access to the tekton-kueue image specified in the kustomization
     
     CI/CD Integration:
         The test runs automatically on pull requests via the GitHub action
         `.github/workflows/test-tekton-kueue-config.yaml` when:
         - Changes are made to `components/kueue/**`
         - The test script itself is modified
         - The workflow file is modified
     
         The test will **FAIL** (not skip) if any prerequisites are missing, ensuring
         issues are caught early in CI/CD pipelines.
     """
     
     import subprocess
     import tempfile
     import os
     import yaml
     import unittest
     from pathlib import Path
    -from typing import Dict, Any, TypedDict
    +from typing import Dict
     from dataclasses import dataclass
     import sys
     
     
     @dataclass
    -class TestConfig:
    +class Prerequisites:
    +    image: str
    +    podman_version: str
     +    config_file: Path
     +    kustomization_file: Path
    -    image: str
    -
    -
    -class ConfigCombination(TypedDict):
    -    config_file: str
    -    kustomization_file: str
    -
    -
    -class TestCombination(TypedDict):
    -    pipelinerun_key: str
    -    config_key: str
    -
    -
    -class PipelineRunMetadata(TypedDict, total=False):
    -    name: str
    -    namespace: str
    -    labels: Dict[str, str]
    -    annotations: Dict[str, str]
    -
    -
    -class PipelineRunDefinition(TypedDict):
    -    apiVersion: str
    -    kind: str
    -    metadata: PipelineRunMetadata
    -    spec: Dict[str, Any]  # More flexible since PipelineRun specs can vary
    -
    -
    -class ExpectedResults(TypedDict):
    -    annotations: Dict[str, str]
    -    labels: Dict[str, str]
    -
    -
    -class PipelineRunTestData(TypedDict):
    -    pipelinerun: PipelineRunDefinition
    -    expected: ExpectedResults
     
     
     def get_tekton_kueue_image(kustomization_file: Path) -> str:
         """Read the tekton-kueue image from the given kustomization file."""
         try:
             with open(kustomization_file, 'r') as f:
                 kustomization = yaml.safe_load(f)
     
             # Look for the tekton-kueue image in the images section
             images = kustomization.get('images', [])
             for image in images:
                 if image.get('name') == 'tekton-kueue':
                     new_name = image.get('newName', '')
                     new_tag = image.get('newTag', '')
                     if new_name and new_tag:
                         return f"{new_name}:{new_tag}"
     
             raise ValueError("tekton-kueue image not found in kustomization")
     
         except Exception as e:
             raise RuntimeError(f"Failed to read tekton-kueue image from {kustomization_file}: {e}")
     
    -def resolve_path(path_str: str, repo_root: Path) -> Path:
    -    """Resolve a path string to an absolute Path, handling both relative and absolute paths."""
    -    if Path(path_str).is_absolute():
    -        return Path(path_str)
    -    return repo_root / path_str
    -
    -
    -def validate_config_combination(config_key: str, repo_root: Path) -> TestConfig:
    -    """Validate and resolve config and kustomization files for a config combination."""
    -    config_data = CONFIG_COMBINATIONS[config_key]
    -
    -    config_file = resolve_path(config_data["config_file"], repo_root)
    -    kustomization_file = resolve_path(config_data["kustomization_file"], repo_root)
    -
    -    # Validate files exist
    +def check_prerequisites(should_print: bool = True) -> Prerequisites:
    +    """Check that all prerequisites are available.
    +
    +    Returns a Prerequisites object with discovered info (image, podman_version)
    +    on success. Raises an exception on failure.
    +    """
    +    messages = ["Checking prerequisites..."]
    +
    +    # Compute repo paths locally
    +    repo_root = Path(__file__).parent.parent
    +    config_file = repo_root / "components/kueue/development/tekton-kueue/config.yaml"
    +    kustomization_file = repo_root / "components/kueue/development/tekton-kueue/kustomization.yaml"
    +
    +    # Config file
     if not config_file.exists():
    -        raise FileNotFoundError(f"Config file not found for config '{config_key}': {config_file}")
    +        raise FileNotFoundError(f"Config file not found: {config_file}")
    +    messages.append(f"\u2713 Config file found: {config_file}")
     
    +    # Kustomization file
     if not kustomization_file.exists():
    -        raise FileNotFoundError(f"Kustomization file not found for config '{config_key}': {kustomization_file}")
    +        raise FileNotFoundError(f"Kustomization file not found: {kustomization_file}")
    +    messages.append(f"\u2713 Kustomization file found: {kustomization_file}")
     
    -    # Get image from kustomization
    +    # Image from kustomization
     image = get_tekton_kueue_image(kustomization_file)
    +    messages.append(f"\u2713 Tekton-kueue image: {image}")
     
    -    return TestConfig(
    -        config_file=config_file,
    -        kustomization_file=kustomization_file,
    -        image=image
    -    )
    -
    -
    -def check_prerequisites(should_print: bool = True) -> Dict[str, TestConfig]:
    -    """Check that all prerequisites are available and pre-process config combinations."""
    -    messages = ["Checking prerequisites..."]
    -    repo_root = Path(__file__).parent.parent
    -
    -    # Check podman availability
    +    # Podman availability
     try:
         result = subprocess.run(["podman", "--version"], capture_output=True, check=True, text=True)
         podman_version = result.stdout.strip()
    -    messages.append(f"✓ Podman available: {podman_version}")
    +    messages.append(f"\u2713 Podman available: {podman_version}")
     except (subprocess.CalledProcessError, FileNotFoundError):
         raise RuntimeError("Podman not available")
     
    -    # Pre-process all unique config combinations
    -    processed_configs: Dict[str, TestConfig] = {}
    -
    -    for _, test_combination in TEST_COMBINATIONS.items():
    -        config_key = test_combination["config_key"]
    -
    -        # Only process each config combination once
    -        if config_key not in processed_configs:
    -            try:
    -                config = validate_config_combination(config_key, repo_root)
    -                processed_configs[config_key] = config
    -                messages.append(f"✓ Config '{config_key}': {config.config_file}, image={config.image}")
    -            except Exception as e:
    -                raise RuntimeError(f"Config '{config_key}' validation failed: {e}")
     
     if should_print:
    -        for message in messages:
    -            print(message)
    +        for line in messages:
    +            print(line)
     
    -    return processed_configs
    +    return Prerequisites(
    +        image=image,
    +        podman_version=podman_version,
    +        config_file=config_file,
    +        kustomization_file=kustomization_file,
    +    )
     
    -# Test PipelineRun definitions (reusable across different configs)
    -PIPELINERUN_DEFINITIONS: Dict[str, PipelineRunTestData] = {
    +# Test PipelineRun definitions
    +TEST_PIPELINERUNS = {
         "multiplatform_new": {
             "name": "Multi-platform pipeline (new style with build-platforms parameter)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-multiplatform-new",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
         },
     
         "multiplatform_old": {
             "name": "Multi-platform pipeline (old style with PLATFORM parameter in tasks)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-multiplatform-old",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "pull_request"
                     }
                 },
                 "spec": {
                     "pipelineSpec": {
                         "tasks": [
                             {
                                 "name": "build-amd64",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/amd64"}]
                             },
                             {
                                 "name": "build-s390x",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/s390x"}]
                             },
                             {
                                 "name": "build-ppc64le",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/ppc64le"}]
                             },
                             {
                                 "name": "build-arm64",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "linux/arm64"}]
                             },
                             {
                                 "name": "build-darwin",
                                 "taskRef": {"name": "build-task"},
                                 "params": [{"name": "PLATFORM", "value": "darwin/amd64"}]
                             }
                         ]
                     },
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-pre-merge-build"
                 }
             }
         },
     
         "release_managed": {
             "name": "Release pipeline (managed)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-release-managed",
                     "namespace": "default",
                     "labels": {
                         "appstudio.openshift.io/service": "release",
                         "pipelines.appstudio.openshift.io/type": "managed"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "release-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-release"
                 }
             }
         },
     
         "release_tenant": {
             "name": "Release pipeline (tenant)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-release-tenant",
                     "namespace": "default",
                     "labels": {
                         "appstudio.openshift.io/service": "release",
                         "pipelines.appstudio.openshift.io/type": "tenant"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "release-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-tenant-release"
                 }
             }
         },
     
         "mintmaker": {
             "name": "Mintmaker pipeline",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-mintmaker",
                     "namespace": "mintmaker"
                 },
                 "spec": {
                     "pipelineRef": {"name": "mintmaker-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-dependency-update"
                 }
             }
         },
     
         "integration_test_push": {
             "name": "Integration test pipeline (push event)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-integration-push",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push",
                         "pipelinesascode.tekton.dev/test-event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "integration-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-test"
                 }
             }
         },
     
         "integration_test_pr": {
             "name": "Integration test pipeline (pull_request event)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-integration-pr",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "pull_request",
                         "pipelinesascode.tekton.dev/test-event-type": "pull_request"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "integration-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-pre-merge-test"
                 }
             }
         },
     
         "default_priority": {
             "name": "Pipeline with default priority",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-default-priority",
                     "namespace": "default"
                 },
                 "spec": {
                     "pipelineRef": {"name": "default-pipeline"},
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {},
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-default"
                 }
             }
         },
     
         "aws_platforms_only": {
             "name": "Pipeline with only AWS-based platforms",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-aws-platforms-only",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["darwin/amd64", "linux/arm64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "2"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
         },
     
         "mixed_platforms_excluded_included": {
             "name": "Pipeline with mixed platforms (some excluded from AWS IP request)",
             "pipelinerun": {
                 "apiVersion": "tekton.dev/v1",
                 "kind": "PipelineRun",
                 "metadata": {
                     "name": "test-mixed-platforms",
                     "namespace": "default",
                     "labels": {
                         "pipelinesascode.tekton.dev/event-type": "push"
                     }
                 },
                 "spec": {
                     "pipelineRef": {"name": "build-pipeline"},
                     "params": [
                         {
                             "name": "build-platforms",
                             "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
                         }
                     ],
                     "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
                 }
             },
             "expected": {
                 "annotations": {
                     "kueue.konflux-ci.dev/requests-linux-amd64": "1",
                     "kueue.konflux-ci.dev/requests-linux-s390x": "1",
                     "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
                     "kueue.konflux-ci.dev/requests-linux-arm64": "1",
                     "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
                     "kueue.konflux-ci.dev/requests-aws-ip": "3"
                 },
                 "labels": {
                     "kueue.x-k8s.io/queue-name": "pipelines-queue",
                     "kueue.x-k8s.io/priority-class": "konflux-post-merge-build"
                 }
             }
    -    },
    -
    -    "user-specific-priority": {
    -        "name": "Multi-platform pipeline with user-specific priority (new style)",
    -        "pipelinerun": {
    -            "apiVersion": "tekton.dev/v1",
    -            "kind": "PipelineRun",
    -            "metadata": {
    -                "name": "test-user-specific-priority",
    -                "namespace": "default",
    -                "labels": {
    -                    "pipelinesascode.tekton.dev/event-type": "push",
    -                    "kueue.x-k8s.io/priority-class": "konflux-user-specific"
    -                }
    -            },
    -            "spec": {
    -                "pipelineRef": {"name": "build-pipeline"},
    -                "params": [
    -                    {
    -                        "name": "build-platforms",
    -                        "value": ["linux/amd64", "linux/s390x", "linux/ppc64le", "linux/arm64", "darwin/amd64"]
    -                    }
    -                ],
    -                "workspaces": [{"name": "shared-workspace", "emptyDir": {}}]
    -            }
    -        },
    -        "expected": {
    -            "annotations": {
    -                "kueue.konflux-ci.dev/requests-linux-amd64": "1",
    -                "kueue.konflux-ci.dev/requests-linux-s390x": "1",
    -                "kueue.konflux-ci.dev/requests-linux-ppc64le": "1",
    -                "kueue.konflux-ci.dev/requests-linux-arm64": "1",
    -                "kueue.konflux-ci.dev/requests-darwin-amd64": "1",
    -                "kueue.konflux-ci.dev/requests-aws-ip": "3"
    -            },
    -            "labels": {
    -                "kueue.x-k8s.io/queue-name": "pipelines-queue",
    -                "kueue.x-k8s.io/priority-class": "konflux-user-specific"
    -            }
    -        }
    -    },
    -
    -}
    -
    -# Configuration combinations that can be applied to any PipelineRun
    -CONFIG_COMBINATIONS: Dict[str, ConfigCombination] = {
    -    "development": {
    -        "name": "Development config",
    -        "config_file": "components/kueue/development/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/development/tekton-kueue/kustomization.yaml"
    -    },
    -    "staging": {
    -        "name": "Staging config",
    -        "config_file": "components/kueue/staging/base/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/staging/base/tekton-kueue/kustomization.yaml"
    -    },
    -    "production": {
    -        "name": "Production config",
    -        "config_file": "components/kueue/production/base/tekton-kueue/config.yaml",
    -        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
    -    },
    -    "production-kflux-ocp-p01": {
    -        "name": "Production config",
    -        "config_file": "components/kueue/production/kflux-ocp-p01/config.yaml",
    -        "kustomization_file": "components/kueue/production/base/tekton-kueue/kustomization.yaml"
    -    }
    -}
    -
    -# Test combinations: which PipelineRuns to test with which configs
    -# This creates a cartesian product of PipelineRuns and configs
    -TEST_COMBINATIONS: Dict[str, TestCombination] = {
    -    # Test all PipelineRuns with development config (default)
    -    "multiplatform_new_dev": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "development"
    -    },
    -    "multiplatform_old_dev": {
    -        "pipelinerun_key": "multiplatform_old",
    -        "config_key": "development"
    -    },
    -    "release_managed_dev": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "development"
    -    },
    -    "release_tenant_dev": {
    -        "pipelinerun_key": "release_tenant",
    -        "config_key": "development"
    -    },
    -    "mintmaker_dev": {
    -        "pipelinerun_key": "mintmaker",
    -        "config_key": "development"
    -    },
    -    "integration_test_push_dev": {
    -        "pipelinerun_key": "integration_test_push",
    -        "config_key": "development"
    -    },
    -    "integration_test_pr_dev": {
    -        "pipelinerun_key": "integration_test_pr",
    -        "config_key": "development"
    -    },
    -    "default_priority_dev": {
    -        "pipelinerun_key": "default_priority",
    -        "config_key": "development"
    -    },
    -    "aws_platforms_only_dev": {
    -        "pipelinerun_key": "aws_platforms_only",
    -        "config_key": "development"
    -    },
    -    "mixed_platforms_excluded_included_dev": {
    -        "pipelinerun_key": "mixed_platforms_excluded_included",
    -        "config_key": "development"
    -    },
    -
    -    # Test key PipelineRuns with staging config
    -    "multiplatform_new_staging": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "staging"
    -    },
    -    "release_managed_staging": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "staging"
    -    },
    -    "integration_test_push_staging": {
    -        "pipelinerun_key": "integration_test_push",
    -        "config_key": "staging"
    -    },
    -
    -    # Test key PipelineRuns with production config
    -    "multiplatform_new_production": {
    -        "pipelinerun_key": "multiplatform_new",
    -        "config_key": "production"
    -    },
    -    "release_managed_production": {
    -        "pipelinerun_key": "release_managed",
    -        "config_key": "production"
    -    },
    -
    -    # Example: Test the same PipelineRun with different configs to show reusability
    -    "user-specific-priority_and_mixed_platforms_production-kflux-ocp-p01": {
    -        "pipelinerun_key": "user-specific-priority",
    -        "config_key": "production-kflux-ocp-p01"
     }
     
     
     class TektonKueueMutationTest(unittest.TestCase):
         """Test suite for tekton-kueue CEL expression mutations."""
     
         @classmethod
         def setUpClass(cls):
             """Set up test class - check prerequisites and pre-process configs."""
    -        cls.processed_configs = check_prerequisites(should_print=False)
    -        cls.repo_root = Path(__file__).parent.parent
    -        print("Prerequisites validated for all tests.")
    +        info = check_prerequisites(should_print=False)
    +        cls.tekton_kueue_image = info.image
    +        cls.config_file = info.config_file
    +        print(f"Using tekton-kueue image: {cls.tekton_kueue_image}")
     
    -    def run_mutation_test(self, test_combination: TestCombination) -> Dict[str, Any]:
    +    def run_mutation_test(self, test_data: Dict) -> Dict:
             """Run a single mutation test and return results."""
    -        # Get pre-processed configuration
    -        config_key = test_combination["config_key"]
    -        test_config = self.processed_configs[config_key]
    -
    -        # Get the PipelineRun definition
    -        pipelinerun_key = test_combination["pipelinerun_key"]
    -        pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
    -        pipelinerun = pipelinerun_data["pipelinerun"]
    +        pipelinerun = test_data["pipelinerun"]
     
             with tempfile.TemporaryDirectory() as temp_dir:
                 # Write the config file
                 config_path = Path(temp_dir) / "config.yaml"
                 pipelinerun_path = Path(temp_dir) / "pipelinerun.yaml"
     
                 # Copy the test-specific config file
                 import shutil
    -            shutil.copy2(test_config.config_file, config_path)
    +            shutil.copy2(self.config_file, config_path)
     
                 # Write the PipelineRun
                 with open(pipelinerun_path, 'w') as f:
                     yaml.dump(pipelinerun, f, default_flow_style=False)
     
                 # Set proper permissions
                 os.chmod(config_path, 0o644)
                 os.chmod(pipelinerun_path, 0o644)
                 os.chmod(temp_dir, 0o755)
     
                 # Run the mutation with test-specific image
                 cmd = [
                     "podman", "run", "--rm",
                     "-v", f"{temp_dir}:/workspace:z",
    -                test_config.image,
    +                self.tekton_kueue_image,
                     "mutate",
                     "--pipelinerun-file", "/workspace/pipelinerun.yaml",
                     "--config-dir", "/workspace"
                 ]
     
                 result = subprocess.run(cmd, capture_output=True, text=True)
     
                 if result.returncode != 0:
                     self.fail(f"Mutation failed: {result.stderr}")
     
                 # Parse the mutated PipelineRun
                 try:
                     mutated = yaml.safe_load(result.stdout)
                 except yaml.YAMLError as e:
                     self.fail(f"Failed to parse mutated YAML: {e}")
     
                 return mutated
     
    -    def validate_mutation_result(self, test_key: str, test_combination: TestCombination) -> None:
    +    def validate_mutation_result(self, test_key: str, test_data: Dict):
             """Helper method to validate mutation results."""
             with self.subTest(test=test_key):
    -            # Get pre-processed configuration for logging
    -            config_key = test_combination["config_key"]
    -            test_config = self.processed_configs[config_key]
    -            print(f"Running test '{test_key}' with config: {test_config.config_file}, image: {test_config.image}")
    -
    -            mutated = self.run_mutation_test(test_combination)
    -
    -            # Get expected results from the PipelineRun definition
    -            pipelinerun_key = test_combination["pipelinerun_key"]
    -            pipelinerun_data = PIPELINERUN_DEFINITIONS[pipelinerun_key]
    -            expected = pipelinerun_data["expected"]
    +            mutated = self.run_mutation_test(test_data)
    +            expected = test_data["expected"]
     
    -            original_metadata = pipelinerun_data["pipelinerun"].get("metadata", {})
    +            original_metadata = test_data["pipelinerun"].get("metadata", {})
                 original_annotations = original_metadata.get("annotations", {}) or {}
                 original_labels = original_metadata.get("labels", {}) or {}
     
                 # Check annotations (full equality vs original + expected)
                 annotations = mutated.get("metadata", {}).get("annotations", {})
                 expected_annotations = expected["annotations"]
                 expected_annotations_full = {**original_annotations, **expected_annotations}
                 self.assertDictEqual(
                     annotations,
                     expected_annotations_full,
                     f"Annotations mismatch; expected {expected_annotations_full}, got {annotations}"
                 )
     
                 # Check labels (full equality vs original + expected)
                 labels = mutated.get("metadata", {}).get("labels", {})
                 expected_labels = expected["labels"]
                 expected_labels_full = {**original_labels, **expected_labels}
                 self.assertDictEqual(
                     labels,
                     expected_labels_full,
                     f"Labels mismatch; expected {expected_labels_full}, got {labels}"
                 )
     
         def test_all_mutations(self):
             """Test all tekton-kueue mutation scenarios."""
    -        for test_key, test_combination in TEST_COMBINATIONS.items():
    -            self.validate_mutation_result(test_key, test_combination)
    +        for test_key, test_data in TEST_PIPELINERUNS.items():
    +            self.validate_mutation_result(test_key, test_data)
     
     
     if __name__ == "__main__":
         import argparse
     
         parser = argparse.ArgumentParser(description="Test tekton-kueue CEL expressions")
         parser.add_argument("--check-setup", action="store_true",
                            help="Check if prerequisites are met and show configuration")
         parser.add_argument("--verbose", "-v", action="store_true",
                            help="Run tests with verbose output")
     
         # Parse known args to allow unittest args to pass through
         args, unknown = parser.parse_known_args()
     
         if args.check_setup:
             try:
    -            processed_configs = check_prerequisites(should_print=True)
    +            info = check_prerequisites(should_print=True)
             except Exception as e:
                 print(f"✗ {e}")
                 sys.exit(1)
     
             print("\n✅ All prerequisites met! Ready to run tests.")
             print("Run: python hack/test-tekton-kueue-config.py")
             print("\nNote: Tests will FAIL (not skip) if any prerequisites are missing.")
     
         else:
             # Run unittest with remaining args

5. kubelet.yaml ignoreDifferences Removal

  • Removing ignoreDifferences and RespectIgnoreDifferences=true from kubelet.yaml could cause Argo CD to report out-of-sync for KubeletConfig resources if there are actual differences that were previously ignored. This change seems unrelated to the PR's main goal and should be carefully evaluated for potential side effects.
    • File: argo-cd-apps/base/all-clusters/infra-deployments/kubelet-config/kubelet.yaml
    --- a/argo-cd-apps/base/all-clusters/infra-deployments/kubelet-config/kubelet.yaml
    +++ b/argo-cd-apps/base/all-clusters/infra-deployments/kubelet-config/kubelet.yaml
    @@ -19,13 +19,6 @@
     metadata:
       name: kubelet-config-{{nameNormalized}}
     spec:
       project: default
       source:
         path: '{{values.sourceRoot}}/{{values.environment}}'
         repoURL: https://github.com/redhat-appstudio/infra-deployments.git
         targetRevision: main
       destination:
         server: '{{server}}'
    -  ignoreDifferences:
    -    - group: machineconfiguration.openshift.io
    -      kind: KubeletConfig
    -      jsonPointers:
    -        - /spec/autoSizingReserved
    -        - /spec/machineConfigPoolSelector
       syncPolicy:
         automated:
           prune: true
           selfHeal: true
         syncOptions:
           - CreateNamespace=false
           - ServerSideApply=true
    -      - RespectIgnoreDifferences=true
         retry:
           limit: -1
           backoff:
             duration: 10s
             factor: 2
             maxDuration: 3m

6. Multi-platform Controller Host Configuration Changes

  • The removal of numerous S390X and D160-M* platforms, their associated configurations, and external secrets in components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml and host-config.yaml represents a significant reduction in supported build platforms. This is a major change that requires explicit confirmation and should ideally be in a separate, well-justified PR.
    • File: components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml
    --- a/components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml
    +++ b/components/multi-platform-controller/production-downstream/kflux-rhel-p01/external-secrets.yaml
    @@ -105,56 +105,10 @@
         - extract:
             key: production/platform/terraform/generated/kflux-rhel-p01/ibm-ppc64le-ssh-key-us-east
       refreshInterval: 1h
       secretStoreRef:
         kind: ClusterSecretStore
         name: appsre-stonesoup-vault
       target:
         creationPolicy: Owner
         deletionPolicy: Delete
         name: ibm-ppc64le-ssh-key
    -----
    -apiVersion: external-secrets.io/v1beta1
    -kind: ExternalSecret
    -metadata:
    -  name: ibm-s390x-ssh-key-regular
    -  namespace: multi-platform-controller
    -  labels:
    -    build.appstudio.redhat.com/multi-platform-secret: "true"
    -  annotations:
    -    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    -    argocd.argoproj.io/sync-wave: "-1"
    -spec:
    -  dataFrom:
    -    - extract:
    -        key: production/platform/terraform/generated/kflux-rhel-p01/ibm-s390x-ssh-key-i6r3
    -  refreshInterval: 1h
    -  secretStoreRef:
    -    kind: ClusterSecretStore
    -    name: appsre-stonesoup-vault
    -  target:
    -    creationPolicy: Owner
    -    deletionPolicy: Delete
    -    name: ibm-s390x-ssh-key-regular
    -----
    -apiVersion: external-secrets.io/v1beta1
    -kind: ExternalSecret
    -metadata:
    -  name: ibm-s390x-ssh-key-large-builder
    -  namespace: multi-platform-controller
    -  labels:
    -    build.appstudio.redhat.com/multi-platform-secret: "true"
    -  annotations:
    -    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    -    argocd.argoproj.io/sync-wave: "-1"
    -spec:
    -  dataFrom:
    -    - extract:
    -        key: production/platform/terraform/generated/kflux-rhel-p01/ibm-s390x-ssh-key-wcrs
    -  refreshInterval: 1h
    -  secretStoreRef:
    -    kind: ClusterSecretStore
    -    name: appsre-stonesoup-vault
    -  target:
    -    creationPolicy: Owner
    -    deletionPolicy: Delete
    -    name: ibm-s390x-ssh-key-large-builder
    • File: components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml
    --- a/components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml
    +++ b/components/multi-platform-controller/production-downstream/kflux-rhel-p01/host-config.yaml
    @@ -9,38 +9,28 @@
       local-platforms: "\
         linux/x86_64,\
         local,\
         localhost,\
         "
       dynamic-platforms: "\
         linux/arm64,\
         linux/amd64,\
         linux-mlarge/arm64,\
         linux-mlarge/amd64,\
    -    linux-d160-mlarge/arm64,\
    -    linux-d160-mlarge/amd64,\
         linux-mxlarge/amd64,\
         linux-mxlarge/arm64,\
    -    linux-d160-mxlarge/arm64,\
    -    linux-d160-mxlarge/amd64,\
         linux-m2xlarge/amd64,\
         linux-m2xlarge/arm64,\
    -    linux-d160-m2xlarge/arm64,\
    -    linux-d160-m2xlarge/amd64,\
         linux-m4xlarge/amd64,\
         linux-m4xlarge/arm64,\
    -    linux-d160-m4xlarge/arm64,\
    -    linux-d160-m4xlarge/amd64,\
         linux-m8xlarge/amd64,\
         linux-m8xlarge/arm64,\
    -    linux-d160-m8xlarge/arm64,\
    -    linux-d160-m8xlarge/amd64,\
         linux-c6gd2xlarge/arm64,\
         linux-cxlarge/amd64,\
         linux-cxlarge/arm64,\
         linux-c2xlarge/amd64,\
         linux-c2xlarge/arm64,\
         linux-c4xlarge/amd64,\
         linux-c4xlarge/arm64,\
         linux-c8xlarge/amd64,\
         linux-c8xlarge/arm64,\
         linux-g6xlarge/amd64,\
    @@ -78,133 +68,68 @@
       dynamic.linux-mlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-mlarge-arm64.instance-type: m6g.large
       dynamic.linux-mlarge-arm64.instance-tag: prod-arm64-mlarge
       dynamic.linux-mlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-mlarge-arm64.aws-secret: aws-account
       dynamic.linux-mlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-mlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-mlarge-arm64.max-instances: "250"
       dynamic.linux-mlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-mlarge-arm64.type: aws
    -  dynamic.linux-d160-mlarge-arm64.region: us-east-1
    -  dynamic.linux-d160-mlarge-arm64.ami: ami-03d6a5256a46c9feb
    -  dynamic.linux-d160-mlarge-arm64.instance-type: m6g.large
    -  dynamic.linux-d160-mlarge-arm64.instance-tag: prod-arm64-mlarge-d160
    -  dynamic.linux-d160-mlarge-arm64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-mlarge-arm64.aws-secret: aws-account
    -  dynamic.linux-d160-mlarge-arm64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-mlarge-arm64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-mlarge-arm64.max-instances: "250"
    -  dynamic.linux-d160-mlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-mlarge-arm64.disk: "160"
    -
       dynamic.linux-mxlarge-arm64.type: aws
       dynamic.linux-mxlarge-arm64.region: us-east-1
       dynamic.linux-mxlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-mxlarge-arm64.instance-type: m6g.xlarge
       dynamic.linux-mxlarge-arm64.instance-tag: prod-arm64-mxlarge
       dynamic.linux-mxlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-mxlarge-arm64.aws-secret: aws-account
       dynamic.linux-mxlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-mxlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-mxlarge-arm64.max-instances: "250"
       dynamic.linux-mxlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-mxlarge-arm64.type: aws
    -  dynamic.linux-d160-mxlarge-arm64.region: us-east-1
    -  dynamic.linux-d160-mxlarge-arm64.ami: ami-03d6a5256a46c9feb
    -  dynamic.linux-d160-mxlarge-arm64.instance-type: m6g.xlarge
    -  dynamic.linux-d160-mxlarge-arm64.instance-tag: prod-arm64-mxlarge-d160
    -  dynamic.linux-d160-mxlarge-arm64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-mxlarge-arm64.aws-secret: aws-account
    -  dynamic.linux-d160-mxlarge-arm64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-mxlarge-arm64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-mxlarge-arm64.max-instances: "250"
    -  dynamic.linux-d160-mxlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-mxlarge-arm64.disk: "160"
    -
       dynamic.linux-m2xlarge-arm64.type: aws
       dynamic.linux-m2xlarge-arm64.region: us-east-1
       dynamic.linux-m2xlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-m2xlarge-arm64.instance-type: m6g.2xlarge
       dynamic.linux-m2xlarge-arm64.instance-tag: prod-arm64-m2xlarge
       dynamic.linux-m2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-m2xlarge-arm64.aws-secret: aws-account
       dynamic.linux-m2xlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-m2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-m2xlarge-arm64.max-instances: "250"
       dynamic.linux-m2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-m2xlarge-arm64.type: aws
    -  dynamic.linux-d160-m2xlarge-arm64.region: us-east-1
    -  dynamic.linux-d160-m2xlarge-arm64.ami: ami-03d6a5256a46c9feb
    -  dynamic.linux-d160-m2xlarge-arm64.instance-type: m6g.2xlarge
    -  dynamic.linux-d160-m2xlarge-arm64.instance-tag: prod-arm64-m2xlarge-d160
    -  dynamic.linux-d160-m2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-m2xlarge-arm64.aws-secret: aws-account
    -  dynamic.linux-d160-m2xlarge-arm64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-m2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-m2xlarge-arm64.max-instances: "250"
    -  dynamic.linux-d160-m2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-m2xlarge-arm64.disk: "160"
    -
       dynamic.linux-m4xlarge-arm64.type: aws
       dynamic.linux-m4xlarge-arm64.region: us-east-1
       dynamic.linux-m4xlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-m4xlarge-arm64.instance-type: m6g.4xlarge
       dynamic.linux-m4xlarge-arm64.instance-tag: prod-arm64-m4xlarge
       dynamic.linux-m4xlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-m4xlarge-arm64.aws-secret: aws-account
       dynamic.linux-m4xlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-m4xlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-m4xlarge-arm64.max-instances: "250"
       dynamic.linux-m4xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-m4xlarge-arm64.type: aws
    -  dynamic.linux-d160-m4xlarge-arm64.region: us-east-1
    -  dynamic.linux-d160-m4xlarge-arm64.ami: ami-03d6a5256a46c9feb
    -  dynamic.linux-d160-m4xlarge-arm64.instance-type: m6g.4xlarge
    -  dynamic.linux-d160-m4xlarge-arm64.instance-tag: prod-arm64-m4xlarge-d160
    -  dynamic.linux-d160-m4xlarge-arm64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-m4xlarge-arm64.aws-secret: aws-account
    -  dynamic.linux-d160-m4xlarge-arm64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-m4xlarge-arm64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-m4xlarge-arm64.max-instances: "250"
    -  dynamic.linux-d160-m4xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-m4xlarge-arm64.disk: "160"
    -
       dynamic.linux-m8xlarge-arm64.type: aws
       dynamic.linux-m8xlarge-arm64.region: us-east-1
       dynamic.linux-m8xlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-m8xlarge-arm64.instance-type: m6g.8xlarge
       dynamic.linux-m8xlarge-arm64.instance-tag: prod-arm64-m8xlarge
       dynamic.linux-m8xlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-m8xlarge-arm64.aws-secret: aws-account
       dynamic.linux-m8xlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-m8xlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-m8xlarge-arm64.max-instances: "250"
       dynamic.linux-m8xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-m8xlarge-arm64.type: aws
    -  dynamic.linux-d160-m8xlarge-arm64.region: us-east-1
    -  dynamic.linux-d160-m8xlarge-arm64.ami: ami-03d6a5256a46c9feb
    -  dynamic.linux-d160-m8xlarge-arm64.instance-type: m6g.8xlarge
    -  dynamic.linux-d160-m8xlarge-arm64.instance-tag: prod-arm64-m8xlarge-d160
    -  dynamic.linux-d160-m8xlarge-arm64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-m8xlarge-arm64.aws-secret: aws-account
    -  dynamic.linux-d160-m8xlarge-arm64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-m8xlarge-arm64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-m8xlarge-arm64.max-instances: "250"
    -  dynamic.linux-d160-m8xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-m8xlarge-arm64.disk: "160"
    -
       dynamic.linux-c6gd2xlarge-arm64.type: aws
       dynamic.linux-c6gd2xlarge-arm64.region: us-east-1
       dynamic.linux-c6gd2xlarge-arm64.ami: ami-03d6a5256a46c9feb
       dynamic.linux-c6gd2xlarge-arm64.instance-type: c6gd.2xlarge
       dynamic.linux-c6gd2xlarge-arm64.instance-tag: prod-arm64-c6gd2xlarge
       dynamic.linux-c6gd2xlarge-arm64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-c6gd2xlarge-arm64.aws-secret: aws-account
       dynamic.linux-c6gd2xlarge-arm64.ssh-secret: aws-ssh-key
       dynamic.linux-c6gd2xlarge-arm64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-c6gd2xlarge-arm64.max-instances: "250"
       dynamic.linux-c6gd2xlarge-arm64.subnet-id: subnet-0f3208c0214c55e2e
    @@ -291,133 +216,68 @@
     
       dynamic.linux-mlarge-amd64.ami: ami-026ebd4cfe2c043b2
       dynamic.linux-mlarge-amd64.instance-type: m7a.large
       dynamic.linux-mlarge-amd64.instance-tag: prod-amd64-mlarge
       dynamic.linux-mlarge-amd64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-mlarge-amd64.aws-secret: aws-account
       dynamic.linux-mlarge-amd64.ssh-secret: aws-ssh-key
       dynamic.linux-mlarge-amd64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-mlarge-amd64.max-instances: "250"
       dynamic.linux-mlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-mlarge-amd64.type: aws
    -  dynamic.linux-d160-mlarge-amd64.region: us-east-1
    -  dynamic.linux-d160-mlarge-amd64.ami: ami-026ebd4cfe2c043b2
    -  dynamic.linux-d160-mlarge-amd64.instance-type: m7a.large
    -  dynamic.linux-d160-mlarge-amd64.instance-tag: prod-amd64-mlarge-d160
    -  dynamic.linux-d160-mlarge-amd64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-mlarge-amd64.aws-secret: aws-account
    -  dynamic.linux-d160-mlarge-amd64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-mlarge-amd64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-mlarge-amd64.max-instances: "250"
    -  dynamic.linux-d160-mlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-mlarge-amd64.disk: "160"
    -
       dynamic.linux-mxlarge-amd64.type: aws
       dynamic.linux-mxlarge-amd64.region: us-east-1
       dynamic.linux-mxlarge-amd64.ami: ami-026ebd4cfe2c043b2
       dynamic.linux-mxlarge-amd64.instance-type: m7a.xlarge
       dynamic.linux-mxlarge-amd64.instance-tag: prod-amd64-mxlarge
       dynamic.linux-mxlarge-amd64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-mxlarge-amd64.aws-secret: aws-account
       dynamic.linux-mxlarge-amd64.ssh-secret: aws-ssh-key
       dynamic.linux-mxlarge-amd64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-mxlarge-amd64.max-instances: "250"
       dynamic.linux-mxlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-mxlarge-amd64.type: aws
    -  dynamic.linux-d160-mxlarge-amd64.region: us-east-1
    -  dynamic.linux-d160-mxlarge-amd64.ami: ami-026ebd4cfe2c043b2
    -  dynamic.linux-d160-mxlarge-amd64.instance-type: m7a.xlarge
    -  dynamic.linux-d160-mxlarge-amd64.instance-tag: prod-amd64-mxlarge-d160
    -  dynamic.linux-d160-mxlarge-amd64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-mxlarge-amd64.aws-secret: aws-account
    -  dynamic.linux-d160-mxlarge-amd64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-mxlarge-amd64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-mxlarge-amd64.max-instances: "250"
    -  dynamic.linux-d160-mxlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-mxlarge-amd64.disk: "160"
    -
       dynamic.linux-m2xlarge-amd64.type: aws
       dynamic.linux-m2xlarge-amd64.region: us-east-1
       dynamic.linux-m2xlarge-amd64.ami: ami-026ebd4cfe2c043b2
       dynamic.linux-m2xlarge-amd64.instance-type: m7a.2xlarge
       dynamic.linux-m2xlarge-amd64.instance-tag: prod-amd64-m2xlarge
       dynamic.linux-m2xlarge-amd64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-m2xlarge-amd64.aws-secret: aws-account
       dynamic.linux-m2xlarge-amd64.ssh-secret: aws-ssh-key
       dynamic.linux-m2xlarge-amd64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-m2xlarge-amd64.max-instances: "250"
       dynamic.linux-m2xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-m2xlarge-amd64.type: aws
    -  dynamic.linux-d160-m2xlarge-amd64.region: us-east-1
    -  dynamic.linux-d160-m2xlarge-amd64.ami: ami-026ebd4cfe2c043b2
    -  dynamic.linux-d160-m2xlarge-amd64.instance-type: m7a.2xlarge
    -  dynamic.linux-d160-m2xlarge-amd64.instance-tag: prod-amd64-m2xlarge-d160
    -  dynamic.linux-d160-m2xlarge-amd64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-m2xlarge-amd64.aws-secret: aws-account
    -  dynamic.linux-d160-m2xlarge-amd64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-m2xlarge-amd64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-m2xlarge-amd64.max-instances: "250"
    -  dynamic.linux-d160-m2xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-m2xlarge-amd64.disk: "160"
    -
       dynamic.linux-m4xlarge-amd64.type: aws
       dynamic.linux-m4xlarge-amd64.region: us-east-1
       dynamic.linux-m4xlarge-amd64.ami: ami-026ebd4cfe2c043b2
       dynamic.linux-m4xlarge-amd64.instance-type: m7a.4xlarge
       dynamic.linux-m4xlarge-amd64.instance-tag: prod-amd64-m4xlarge
       dynamic.linux-m4xlarge-amd64.key-name: kflux-rhel-p01-key-pair
       dynamic.linux-m4xlarge-amd64.aws-secret: aws-account
       dynamic.linux-m4xlarge-amd64.ssh-secret: aws-ssh-key
       dynamic.linux-m4xlarge-amd64.security-group-id: sg-0c67a834068be63d6
       dynamic.linux-m4xlarge-amd64.max-instances: "250"
       dynamic.linux-m4xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
     
    -  dynamic.linux-d160-m4xlarge-amd64.type: aws
    -  dynamic.linux-d160-m4xlarge-amd64.region: us-east-1
    -  dynamic.linux-d160-m4xlarge-amd64.ami: ami-026ebd4cfe2c043b2
    -  dynamic.linux-d160-m4xlarge-amd64.instance-type: m7a.4xlarge
    -  dynamic.linux-d160-m4xlarge-amd64.instance-tag: prod-amd64-m4xlarge-d160
    -  dynamic.linux-d160-m4xlarge-amd64.key-name: kflux-rhel-p01-key-pair
    -  dynamic.linux-d160-m4xlarge-amd64.aws-secret: aws-account
    -  dynamic.linux-d160-m4xlarge-amd64.ssh-secret: aws-ssh-key
    -  dynamic.linux-d160-m4xlarge-amd64.security-group-id: sg-0c67a834068be63d6
    -  dynamic.linux-d160-m4xlarge-amd64.max-instances: "250"
    -  dynamic.linux-d160-m4xlarge-amd64.subnet-id: subnet-0f3208c0214c55e2e
    -  dynamic.linux-d160-m4xlarge-amd64.disk: "160"
    -
       dynamic.linux-m8xlarge-amd64.type: aws
       dynamic.linux-m8xlarge-amd64.region: us-east-1
       dynamic.linux-m8xlarge-amd64.ami: ami-026ebd4cfe2c043b2
       dynamic.linux-m8xlarge-amd64.instance-type: m7a.8xlarge
       dynamic.linux-m8xlarge-amd64.instance-tag: prod-amd64-m8xlarge
       dynamic.linux-m8xlarge-amd64.key-name: kflu...*[Comment body truncated]*

Copy link
Contributor

Code Review by Gemini

### `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml`

**Issue:** The `ApplicationSet` hardcodes the `environment` to `development`, which prevents deployment to `staging` environments as stated in the commit message.

**Suggestion:** Dynamically set the `environment` based on a label on the target cluster. This assumes that member clusters are labeled with `appstudio.redhat.com/environment` (e.g., `development`, `staging`).

```diff
--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -12,7 +12,7 @@
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
-                environment: development
+                environment: '{{metadata.labels.appstudio.redhat.com/environment}}'
                 clusterDir: ""
           - list:
               elements: []

Issue: The list generator with elements: [] is empty and does not contribute any values to the merge operation, making it redundant.

Suggestion: Remove the empty list generator for clarity.

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -14,8 +14,6 @@
               values:
                 sourceRoot: components/squid
                 environment: '{{metadata.labels.appstudio.redhat.com/environment}}'
                 clusterDir: ""
-          - list:
-              elements: []
   template:
     metadata:
       name: squid-{{nameNormalized}}

components/squid/development/squid-helm-generator.yaml

Issue: The valuesInline.environment is set to release, which might be inconsistent with the development environment context.

Suggestion: Align the environment value passed to the Helm chart with the actual deployment environment for better clarity and potential environment-specific configurations within the chart.

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -12,4 +12,4 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:
       cpu: 100m
       memory: 128Mi

components/squid/staging/squid-helm-generator.yaml

Issue: The valuesInline.environment is set to release, which might be inconsistent with the staging environment context.

Suggestion: Align the environment value passed to the Helm chart with the actual deployment environment for better clarity and potential environment-specific configurations within the chart.

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -12,4 +12,4 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:
       cpu: 100m
       memory: 128Mi

Copy link
Contributor

Code Review by Gemini

Here are some observations and suggested improvements for the changes:

**1. `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml` - ApplicationSet configuration**

The current `ApplicationSet` configuration in `squid.yaml` hardcodes `environment: staging` for all member clusters. This means the `source.path` will always resolve to `components/squid/staging/`, effectively preventing the `components/squid/development/` configuration from being used, even for clusters intended for development. This contradicts the commit message "Deploy squid to development and staging environments".

To correctly deploy to both development and staging environments, the `ApplicationSet` needs to dynamically select the environment based on cluster labels.

**Suggested change:**

```diff
--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -9,14 +9,24 @@
         mergeKeys:
           - nameNormalized
         generators:
           - clusters:
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
+                  appstudio.redhat.com/environment: development
+              values:
+                sourceRoot: components/squid
+                environment: development
+                clusterDir: ""
+          - clusters:
+              selector:
+                matchLabels:
+                  appstudio.redhat.com/member-cluster: "true"
+                  appstudio.redhat.com/environment: staging
+              values:
+                sourceRoot: components/squid
+                environment: staging
                 clusterDir: ""
           - list:
               elements: []
   template:

2. components/squid/development/squid-helm-generator.yaml - Helm Chart Environment Value

The valuesInline.environment is set to release in the development Helm generator. If the Helm chart uses this value for environment-specific logic, it should reflect the actual environment (e.g., development).

Suggested change:

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:
       cpu: 100m

3. components/squid/staging/squid-helm-generator.yaml - Helm Chart Environment Value

Similar to the development environment, the valuesInline.environment is set to release in the staging Helm generator. This should be updated to staging if the Helm chart uses this value for environment-specific logic.

Suggested change:

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:
       cpu: 100m

4. argo-cd-apps/overlays/development/kustomization.yaml - Redundant Patch

With the ApplicationSet in argo-cd-apps/base/member/infra-deployments/squid/squid.yaml now dynamically selecting the environment based on cluster labels, the patch for squid in the development overlay becomes redundant and can be removed. The base ApplicationSet will correctly handle deployments to development clusters.

Suggested change:

--- a/argo-cd-apps/overlays/development/kustomization.yaml
+++ b/argo-cd-apps/overlays/development/kustomization.yaml
@@ -227,7 +227,4 @@
     target:
       kind: ApplicationSet
       version: v1alpha1
       name: trust-manager
-  - path: development-overlay-patch.yaml
-    target:
-      kind: ApplicationSet
-      version: v1alpha1
-      name: squid

@avi-biton
Copy link
Contributor Author

@yftacherzog @ifireball @gbenhaim @amisstea PTAL :)

@avi-biton
Copy link
Contributor Author

/test appstudio-e2e-tests

Copy link
Contributor

Code Review by Gemini

Here are some observations and suggested improvements for the changes:

**1. `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml` - ApplicationSet configuration**

The current `ApplicationSet` configuration in `squid.yaml` hardcodes `environment: staging` for all member clusters. This means the `source.path` will always resolve to `components/squid/staging/`, effectively preventing the `components/squid/development/` configuration from being used, even for clusters intended for development. This contradicts the commit message "Deploy squid to development and staging environments".

To correctly deploy to both development and staging environments, the `ApplicationSet` needs to dynamically select the environment based on cluster labels.

**Suggested change:**

```diff
--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -9,14 +9,24 @@
         mergeKeys:
           - nameNormalized
         generators:
           - clusters:
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
+                  appstudio.redhat.com/environment: development
+              values:
+                sourceRoot: components/squid
+                environment: development
+                clusterDir: ""
+          - clusters:
+              selector:
+                matchLabels:
+                  appstudio.redhat.com/member-cluster: "true"
+                  appstudio.redhat.com/environment: staging
+              values:
+                sourceRoot: components/squid
+                environment: staging
                 clusterDir: ""
           - list:
               elements: []
   template:

2. components/squid/development/squid-helm-generator.yaml - Helm Chart Environment Value

The valuesInline.environment is set to release in the development Helm generator. If the Helm chart uses this value for environment-specific logic, it should reflect the actual environment (e.g., development).

Suggested change:

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:
       cpu: 100m

3. components/squid/staging/squid-helm-generator.yaml - Helm Chart Environment Value

Similar to the development environment, the valuesInline.environment is set to release in the staging Helm generator. This should be updated to staging if the Helm chart uses this value for environment-specific logic.

Suggested change:

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
   test:
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:
       cpu: 100m

4. argo-cd-apps/overlays/development/kustomization.yaml - Redundant Patch

With the ApplicationSet in argo-cd-apps/base/member/infra-deployments/squid/squid.yaml now dynamically selecting the environment based on cluster labels, the patch for squid in the development overlay becomes redundant and can be removed. The base ApplicationSet will correctly handle deployments to development clusters.

Suggested change:

--- a/argo-cd-apps/overlays/development/kustomization.yaml
+++ b/argo-cd-apps/overlays/development/kustomization.yaml
@@ -227,7 +227,4 @@
     target:
       kind: ApplicationSet
       version: v1alpha1
       name: trust-manager
-  - path: development-overlay-patch.yaml
-    target:
-      kind: ApplicationSet
-      version: v1alpha1
-      name: squid

Copy link
Contributor

Code Review by Gemini

The following issues and improvements have been identified:

Bug

File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml

The ApplicationSet is configured to deploy Squid to all member clusters, but the clusters generator hardcodes the environment value to staging. This means that even development clusters will attempt to deploy the staging configuration, contradicting the commit message "deploy squid to development and staging environments" and the existence of separate development and staging configurations under components/squid.

Suggested change:

To correctly deploy different configurations based on the cluster's environment, the generators section should include separate clusters generators, each targeting specific environment labels (e.g., environment: development or environment: staging) and setting the corresponding environment value.

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -6,15 +6,26 @@
     - merge:
         mergeKeys:
           - nameNormalized
         generators:
           - clusters:
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
+                  environment: development # Target development clusters
+              values:
+                sourceRoot: components/squid
+                environment: development # Use development configuration
                 clusterDir: ""
+          - clusters:
+              selector:
+                matchLabels:
+                  appstudio.redhat.com/member-cluster: "true"
+                  environment: staging # Target staging clusters
+              values:
+                sourceRoot: components/squid
+                environment: staging # Use staging configuration
+                clusterDir: ""
           - list:
               elements: []
   template:
     metadata:

Improvement

Files: components/squid/development/squid-helm-generator.yaml and components/squid/staging/squid-helm-generator.yaml

The valuesInline.environment field in both HelmChartInflationGenerator files is set to release. If the Helm chart for Squid uses this environment value internally for environment-specific logic or configuration, it should be aligned with the actual environment (e.g., development for development and staging for staging). This ensures consistency and correct behavior within the Helm chart itself.

Suggested changes:

File: components/squid/development/squid-helm-generator.yaml

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -14,4 +14,4 @@
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:

File: components/squid/staging/squid-helm-generator.yaml

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -14,4 +14,4 @@
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:

Copy link
Contributor

@amisstea amisstea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/retest
/lgtm

repoURL: https://github.com/redhat-appstudio/infra-deployments.git
targetRevision: main
destination:
namespace: proxy
Copy link
Member

@gbenhaim gbenhaim Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better to call the namespace konflux-caching-proxy just so it will be clear what is the purpose of this proxy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ADR mentions the proxy will be available at http.proxy.svc.cluster.local (https://github.com/konflux-ci/architecture/blob/3f21424a9a089347fee6185e6763c62f1c3f51d8/ADR/0047-caching-for-container-build-layers.md#L28) which means deploying to the proxy namespace. On the other hand, our service name is currently squid and not http. We should reconcile these one way or another. @ifireball do you have a strong preference for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might update the namespace before we will bump to production

Copy link
Contributor

Code Review by Gemini

The following issues have been identified in the provided code changes:

Issue 1: Hardcoded environment in ApplicationSet

The ApplicationSet for squid hardcodes the environment variable to staging within its clusters generator. This means that all member clusters will deploy the staging configuration, contradicting the commit message's intent to deploy to both development and staging environments. To correctly deploy environment-specific configurations, the environment should be dynamically determined, typically from cluster labels.

File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
Suggested change:

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -12,7 +12,7 @@
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
-                environment: staging
+                environment: '{{ (index .metadata.labels "appstudio.redhat.com/environment") | default "staging" }}'
                 clusterDir: ""
           - list:
               elements: []

This change assumes that member clusters are labeled with appstudio.redhat.com/environment (e.g., development, staging). If a cluster does not have this label, it will default to staging.

Issue 2: Incorrect environment in HelmChartInflationGenerator for development

The HelmChartInflationGenerator for the development environment specifies environment: release in its valuesInline. This should be development to correctly reflect the environment.

File: components/squid/development/squid-helm-generator.yaml
Suggested change:

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:
       cpu: 100m

Issue 3: Incorrect environment in HelmChartInflationGenerator for staging

Similar to the development environment, the HelmChartInflationGenerator for the staging environment also specifies environment: release in its valuesInline. This should be staging.

File: components/squid/staging/squid-helm-generator.yaml
Suggested change:

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -15,7 +15,7 @@
     enabled: false
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:
       cpu: 100m

Copy link
Contributor

Code Review by Gemini

The following issues and improvements are identified in the provided code changes:

Bug

File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml

The ApplicationSet is configured to deploy the staging environment configuration to all member clusters, regardless of whether they are development or staging clusters. This contradicts the commit message "Deploy squid to development and staging environments" which implies different configurations for different environments. To deploy distinct configurations for development and staging environments, the ApplicationSet needs to use separate generators with appropriate cluster selectors and environment values.

Exact lines to change:

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -4,19 +4,29 @@
   generators:
     - merge:
         mergeKeys:
           - nameNormalized
         generators:
           - clusters:
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
+                  environment: development # Assuming clusters are labeled with 'environment: development'
+              values:
+                sourceRoot: components/squid
+                environment: development
+                clusterDir: ""
+          - clusters:
+              selector:
+                matchLabels:
+                  appstudio.redhat.com/member-cluster: "true"
+                  environment: staging # Assuming clusters are labeled with 'environment: staging'
+              values:
+                sourceRoot: components/squid
+                environment: staging
                 clusterDir: ""
           - list:
               elements: []
   template:
     metadata:
       name: squid-{{nameNormalized}}
       annotations:

Improvement

Files: components/squid/development/squid-helm-generator.yaml, components/squid/staging/squid-helm-generator.yaml

Both squid-helm-generator.yaml files set valuesInline.environment: release. If the Helm chart consumes this environment value for environment-specific logic, it should be set to development in the development overlay and staging in the staging overlay to ensure correct environment-specific behavior within the Helm chart. If the Helm chart does not use this value, it can be removed or left as is, but aligning it with the actual environment is generally a good practice for clarity.

Exact lines to change for components/squid/development/squid-helm-generator.yaml:

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -14,4 +14,4 @@
   cert-manager:
     enabled: false
-  environment: release
+  environment: development
   resources:
     requests:

Exact lines to change for components/squid/staging/squid-helm-generator.yaml:

--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -14,4 +14,4 @@
   cert-manager:
     enabled: false
-  environment: release
+  environment: staging
   resources:
     requests:

Copy link
Contributor

Code Review by Gemini

The following issues and improvements have been identified:

Issue 1: ApplicationSet hardcodes staging environment for all member clusters

The ApplicationSet for squid is configured to deploy the staging environment configuration to all clusters matching appstudio.redhat.com/member-cluster: "true". This means development clusters will also receive the staging configuration, which contradicts the intent of deploying to both development and staging environments with potentially different configurations, as implied by the separate components/squid/development and components/squid/staging directories.

File: argo-cd-apps/base/member/infra-deployments/squid/squid.yaml

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -6,19 +6,29 @@
     - merge:
         mergeKeys:
           - nameNormalized
         generators:
           - clusters:
               selector:
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
-                clusterDir: ""
+                  appstudio.redhat.com/environment: "development"
+              values:
+                sourceRoot: components/squid
+                environment: development
+                clusterDir: ""
+          - clusters:
+              selector:
+                matchLabels:
+                  appstudio.redhat.com/member-cluster: "true"
+                  appstudio.redhat.com/environment: "staging"
+              values:
+                sourceRoot: components/squid
+                environment: staging
+                clusterDir: ""
           - list:
               elements: []
   template:
     metadata:
       name: squid-{{nameNormalized}}
       annotations:
         argocd.argoproj.io/sync-wave: "1"

Improvement 1: Identical Helm generator configurations for development and staging

The squid-helm-generator.yaml files in both components/squid/development and components/squid/staging are currently identical. If there are no planned differences between the development and staging configurations for Squid, consider consolidating these into a single base configuration (e.g., components/squid/base) to reduce redundancy. If future differences are expected, the current structure is acceptable.

Files:

  • components/squid/development/squid-helm-generator.yaml
  • components/squid/staging/squid-helm-generator.yaml

No specific line changes are suggested for this point, but it's a structural consideration.

Copy link
Contributor

Code Review by Gemini

### `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml`

**Bug:** The `merge` generator will not produce any applications because one of its sub-generators, the `list` generator, is empty. For the `merge` generator to produce output, all its sub-generators must produce at least one element.

```diff
--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -3,19 +3,14 @@
   name: squid
 spec:
   generators:
-    - merge:
-        mergeKeys:
-          - nameNormalized
-        generators:
-          - clusters:
-              selector:
-                matchLabels:
-                  appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
-                clusterDir: ""
-          - list:
-              elements: []
+    - clusters:
+        selector:
+          matchLabels:
+            appstudio.redhat.com/member-cluster: "true"
+        values:
+          sourceRoot: components/squid
+          environment: staging
+          clusterDir: ""
   template:
     metadata:
       name: squid-{{nameNormalized}}

components/squid/development/squid-helm-generator.yaml

components/squid/staging/squid-helm-generator.yaml

Improvement: The installCertManagerComponents: false and cert-manager: enabled: false values might be redundant depending on the Helm chart's values.yaml. If installCertManagerComponents is a top-level value that controls cert-manager installation, then cert-manager.enabled: false might be unnecessary. Please verify the Helm chart's values.yaml to confirm if both are required.

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:

Improvement: The files components/squid/development/squid-helm-generator.yaml and components/squid/staging/squid-helm-generator.yaml are currently identical. If there are no planned differences between development and staging configurations for Squid, consider consolidating these into a single base configuration to reduce duplication.

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:
--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:

Copy link
Contributor

Code Review by Gemini

### `argo-cd-apps/base/member/infra-deployments/squid/squid.yaml`

**Bug:** The `merge` generator will not produce any applications because one of its sub-generators, the `list` generator, is empty. For the `merge` generator to produce output, all its sub-generators must produce at least one element.

```diff
--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -3,19 +3,14 @@
   name: squid
 spec:
   generators:
-    - merge:
-        mergeKeys:
-          - nameNormalized
-        generators:
-          - clusters:
-              selector:
-                matchLabels:
-                  appstudio.redhat.com/member-cluster: "true"
-              values:
-                sourceRoot: components/squid
-                environment: staging
-                clusterDir: ""
-          - list:
-              elements: []
+    - clusters:
+        selector:
+          matchLabels:
+            appstudio.redhat.com/member-cluster: "true"
+        values:
+          sourceRoot: components/squid
+          environment: staging
+          clusterDir: ""
   template:
     metadata:
       name: squid-{{nameNormalized}}

components/squid/development/squid-helm-generator.yaml

components/squid/staging/squid-helm-generator.yaml

Improvement: The installCertManagerComponents: false and cert-manager: enabled: false values might be redundant depending on the Helm chart's values.yaml. If installCertManagerComponents is a top-level value that controls cert-manager installation, then cert-manager.enabled: false might be unnecessary. Please verify the Helm chart's values.yaml to confirm if both are required.

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:

Improvement: The files components/squid/development/squid-helm-generator.yaml and components/squid/staging/squid-helm-generator.yaml are currently identical. If there are no planned differences between development and staging configurations for Squid, consider consolidating these into a single base configuration to reduce duplication.

--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:
--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -10,8 +10,7 @@
 valuesInline:
   installCertManagerComponents: false
   mirrord:
     enabled: false
   test:
     enabled: false
-  cert-manager:
-    enabled: false
   environment: release
   resources:
     requests:

Copy link
Contributor

ERROR: # Code Review by Gemini
An unexpected error occurred in Client call: Server disconnected without sending a response.

Copy link
Contributor

Code Review by Gemini

The changes introduce several new components related to the "sandbox" and "toolchain" operators, along with a new squid proxy. While the commit message specifically mentions deploying squid to development and staging and excluding it from production, the other new components appear to have inconsistent deployment configurations, leading to potential issues.

Here are the identified issues and suggested improvements:

1. Inconsistent Deployment Scope for New "Sandbox" Components

The commit message explicitly states that squid should be deployed to development and staging environments and excluded from production. However, other newly introduced "sandbox" components (kubesaw-common, toolchain-host-operator, toolchain-member-operator, ui) are configured in a way that contradicts this, or lacks explicit exclusion for production.

  • Issue: The ApplicationSets for kubesaw-common, toolchain-host-operator, toolchain-member-operator, and ui are configured to deploy to clusters that include production environments (either implicitly by matching all clusters or explicitly in the elements list), but they hardcode environment: staging in their source.path. This means production clusters will attempt to deploy staging configurations, which is incorrect and could lead to unexpected behavior or failures. Furthermore, unlike squid, there are no corresponding delete patches in the production overlays for these components.
  • Recommendation: Clarify the intended deployment scope for kubesaw-common, toolchain-host-operator, toolchain-member-operator, and ui.
    • If these components are intended for staging/development environments only:
      • Add explicit delete patches for these ApplicationSets in the konflux-public-production/delete-applications.yaml and production-downstream/delete-applications.yaml overlays, similar to how squid is handled.
      • For kubesaw-common.yaml, add a selector to the clusters generator to target only staging/development clusters.
      • For toolchain-host-operator.yaml, toolchain-member-operator.yaml, and ui.yaml, ensure their elements lists only contain staging/development clusters.
    • If these components are intended for all environments (staging and production) with environment-specific configurations:
      • Modify the source.path in each ApplicationSet to dynamically use the cluster's environment label (e.g., path: '{{values.sourceRoot}}/{{labels.environment}}/{{values.clusterDir}}'). This would require ensuring all clusters have an environment label set to staging or production.

Given the commit message's focus on squid for staging/development only, the first option (staging/development only deployment with production exclusions) seems more aligned with the stated intent.

2. Inconsistent Namespace Definitions for Monitoring Resources

Several new monitoring-related resources explicitly define namespace: system or namespace: appstudio-workload-monitoring in their metadata, while their parent kustomization.yaml files specify namespace: toolchain-host-operator or namespace: toolchain-member-operator. This inconsistency will lead to resources being created in unintended namespaces or deployment failures.

  • Issue 1: components/sandbox/toolchain-host-operator/base/monitoring/prometheus-network-policy.yaml

    --- a/components/sandbox/toolchain-host-operator/base/monitoring/prometheus-network-policy.yaml
    +++ /dev/null
    @@ -0,0 +1,14 @@
    +apiVersion: networking.k8s.io/v1
    +kind: NetworkPolicy
    +metadata:
    +  name: allow-from-openshift-user-workload-monitoring
    +  namespace: system
    +spec:
    +  ingress:
    +    - from:
    +        - namespaceSelector:
    +            matchLabels:
    +              kubernetes.io/metadata.name: openshift-user-workload-monitoring
    +  podSelector: {}
    +  policyTypes:
    +    - Ingress
    • Problem: This NetworkPolicy explicitly sets namespace: system. However, the kustomization.yaml for components/sandbox/toolchain-host-operator/base/monitoring sets namespace: toolchain-host-operator. This means the NetworkPolicy will be deployed to the toolchain-host-operator namespace, but it's configured to apply to pods in the system namespace, which is incorrect.
    • Recommendation: If this NetworkPolicy is intended for the toolchain-host-operator namespace, remove the namespace: system line from the metadata section. If it's truly intended for the system namespace, it should be moved to a kustomization that targets the system namespace.
  • Issue 2: components/sandbox/toolchain-host-operator/base/monitoring/serviceaccount.yaml

    --- a/components/sandbox/toolchain-host-operator/base/monitoring/serviceaccount.yaml
    +++ /dev/null
    @@ -0,0 +1,14 @@
    +apiVersion: v1
    +kind: Secret
    +metadata:
    +  annotations:
    +    kubernetes.io/service-account.name: host-operator-metrics-reader
    +  name: host-operator-metrics-reader
    +  namespace: system
    +type: kubernetes.io/service-account-token
    +---
    +apiVersion: v1
    +kind: ServiceAccount
    +metadata:
    +  name: host-operator-metrics-reader
    +  namespace: system
    • Problem: The ServiceAccount and Secret explicitly set namespace: system. Similar to the NetworkPolicy, this conflicts with the parent kustomization.yaml which sets namespace: toolchain-host-operator.
    • Recommendation: Remove the namespace: system lines from the metadata sections of both the ServiceAccount and Secret. This will allow Kustomize to apply the correct namespace (toolchain-host-operator).
  • Issue 3: components/sandbox/toolchain-host-operator/base/monitoring/sandbox-registration-service-proxy.yaml

    --- a/components/sandbox/toolchain-host-operator/base/monitoring/sandbox-registration-service-proxy.yaml
    +++ /dev/null
    @@ -0,0 +1,56 @@
    +apiVersion: v1
    +kind: ServiceAccount
    +metadata:
    +  name: registration-service-metrics-reader
    +  namespace: system
    +  annotations:
    +    kubernetes.io/service-account.name: registration-service-metrics-reader
    +type: kubernetes.io/service-account-token
    +---
    +apiVersion: rbac.authorization.k8s.io/v1
    +kind: ClusterRole
    +metadata:
    +  name: registration-service-metrics-reader
    +rules:
    +- nonResourceURLs:
    +  - /metrics
    +  verbs:
    +  - get
    +---
    +apiVersion: rbac.authorization.k8s.io/v1
    +kind: ClusterRoleBinding
    +metadata:
    +  name: prometheus-registration-service-metrics-reader
    +roleRef:
    +  apiGroup: rbac.authorization.k8s.io
    +  kind: ClusterRole
    +  name: registration-service-metrics-reader
    +subjects:
    +- kind: ServiceAccount
    +  name: registration-service-metrics-reader
    +  namespace: system
    +---
    +apiVersion: monitoring.coreos.com/v1
    +kind: ServiceMonitor
    +metadata:
    +  name: sandbox-registration-service-proxy
    +  namespace: system
    +spec:
    +  endpoints:
    +    - interval: 15s
    +      scheme: http
    +      path: /metrics
    +      port: proxy-metrics
    +      authorization:
    +        credentials:
    +          key: token
    +          name: "registration-service-metrics-reader"
    +  selector:
    +    matchLabels:
    +      run: proxy-metrics
    • Problem: The ServiceAccount, Secret, ClusterRoleBinding subject, and ServiceMonitor all explicitly set namespace: system. This conflicts with the parent kustomization.yaml which sets namespace: toolchain-host-operator.
    • Recommendation: Remove the namespace: system lines from the metadata sections of these resources. This will allow Kustomize to apply the correct namespace (toolchain-host-operator).
  • Issue 4: components/sandbox/toolchain-member-operator/base/monitoring/serviceaccount.yaml

    --- a/components/sandbox/toolchain-member-operator/base/monitoring/serviceaccount.yaml
    +++ /dev/null
    @@ -0,0 +1,14 @@
    +apiVersion: v1
    +kind: Secret
    +metadata:
    +  annotations:
    +    kubernetes.io/service-account.name: member-operator-metrics-reader
    +  name: member-operator-metrics-reader
    +  namespace: system
    +type: kubernetes.io/service-account-token
    +---
    +apiVersion: v1
    +kind: ServiceAccount
    +metadata:
    +  name: member-operator-metrics-reader
    +  namespace: system
    • Problem: The ServiceAccount and Secret explicitly set namespace: system. This conflicts with the parent kustomization.yaml which sets namespace: toolchain-member-operator.
    • Recommendation: Remove the namespace: system lines from the metadata sections of both the ServiceAccount and Secret. This will allow Kustomize to apply the correct namespace (toolchain-member-operator).
  • Issue 5: components/sandbox/toolchain-member-operator/base/monitoring/rbac.yaml

    --- a/components/sandbox/toolchain-member-operator/base/monitoring/rbac.yaml
    +++ /dev/null
    @@ -0,0 +1,23 @@
    +---
    +apiVersion: rbac.authorization.k8s.io/v1
    +kind: ClusterRole
    +metadata:
    +  name: member-operator-metrics
    +rules:
    +- nonResourceURLs:
    +  - /metrics
    +  verbs:
    +  - get
    +---
    +apiVersion: rbac.authorization.k8s.io/v1
    +kind: ClusterRoleBinding
    +metadata:
    +  name: prometheus-member-operator-metrics-reader
    +roleRef:
    +  apiGroup: rbac.authorization.k8s.io
    +  kind: ClusterRole
    +  name: member-operator-metrics
    +subjects:
    +- kind: ServiceAccount
    +  name: member-operator-metrics-reader
    +  namespace: system
    • Problem: The ClusterRoleBinding subject explicitly sets namespace: system. This conflicts with the parent kustomization.yaml which sets namespace: toolchain-member-operator.
    • Recommendation: Change namespace: system to namespace: toolchain-member-operator in the subjects section of the ClusterRoleBinding to match the intended deployment namespace.

3. Missing Namespace Creation for New Operators

The ApplicationSets for toolchain-host-operator and toolchain-member-operator set syncPolicy.syncOptions: - CreateNamespace=false. This implies that their respective namespaces (toolchain-host-operator and toolchain-member-operator) must already exist. However, these namespaces are defined in components/sandbox/toolchain-host-operator/base/ns.yaml and components/sandbox/toolchain-member-operator/base/ns.yaml with name: to-be-added-by-kustomize.

  • Issue: The CreateNamespace=false setting combined with a placeholder namespace name in ns.yaml and the fact that these are new components means the namespaces might not be created, leading to deployment failures.

  • Recommendation: Change CreateNamespace=false to CreateNamespace=true in the ApplicationSets for toolchain-host-operator and toolchain-member-operator to ensure the namespaces are created if they don't exist.

    • File: argo-cd-apps/base/host/toolchain-host-operator/toolchain-host-operator.yaml

      --- a/argo-cd-apps/base/host/toolchain-host-operator/toolchain-host-operator.yaml
      +++ b/argo-cd-apps/base/host/toolchain-host-operator/toolchain-host-operator.yaml
      @@ -40,7 +40,7 @@
       server: '{{server}}'
       syncPolicy:
       automated:
       prune: false
       selfHeal: true
       syncOptions:
      -    - CreateNamespace=false
      +    - CreateNamespace=true
       retry:
       limit: -1
       backoff:
    • File: argo-cd-apps/base/toolchain-member/toolchain-member-operator.yaml

      --- a/argo-cd-apps/base/toolchain-member/toolchain-member-operator.yaml
      +++ b/argo-cd-apps/base/toolchain-member/toolchain-member-operator.yaml
      @@ -50,7 +50,7 @@
       server: '{{server}}'
       syncPolicy:
       automated:
       prune: false
       selfHeal: true
       syncOptions:
      -    - CreateNamespace=false
      +    - CreateNamespace=true
       retry:
       limit: -1
       backoff:

Copy link
Contributor

Code Review by Gemini

--- a/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
+++ b/argo-cd-apps/base/member/infra-deployments/squid/squid.yaml
@@ -10,7 +10,7 @@
                 matchLabels:
                   appstudio.redhat.com/member-cluster: "true"
               values:
                 sourceRoot: components/squid
-                environment: staging
+                environment: '{{labels.environment}}'
                 clusterDir: ""
           - list:
               elements: []
@@ -27,7 +27,7 @@
       syncPolicy:
         automated:
           prune: true
-          selfHeal: false
+          selfHeal: true
         syncOptions:
           - CreateNamespace=true
         retry:

Issue: The ApplicationSet hardcodes the environment to staging, meaning all member clusters (including development ones) will deploy the staging configuration of Squid. Additionally, selfHeal is set to false, which is generally not desired for infrastructure components.

Suggestion:

  1. Use {{labels.environment}} to dynamically select the environment based on the cluster's environment label. This allows development clusters to use components/squid/development and staging clusters to use components/squid/staging.
  2. Set selfHeal: true to ensure Argo CD automatically corrects any drifts from the desired state.

--- a/components/release/development/kustomization.yaml
+++ b/components/release/development/kustomization.yaml
@@ -13,10 +13,6 @@
 
 namespace: release-service
 
 patches:
   - target:
       group: apps
       version: v1
       kind: Deployment
       name: controller-manager
     path: sealights-token-patch.yaml
-  - target:
-      group: apps
-      version: v1
-      kind: Deployment
-      name: release-service-controller-manager
-    path: manager-memory-patch.yaml
--- /dev/null
+++ b/components/release/development/manager-memory-patch.yaml
@@ -0,0 +1,13 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: release-service-controller-manager
+  namespace: system
+spec:
+  template:
+    spec:
+      containers:
+      - name: manager
+        resources:
+          limits:
+            memory: 256Mi

Issue: The changes related to release-service memory limits are unrelated to the commit message "deploy squid to development and staging". Including unrelated changes in a pull request makes it harder to review and track.

Suggestion: Revert these changes and submit them in a separate pull request focused solely on the release-service memory limits.


--- a/components/squid/development/squid-helm-generator.yaml
+++ b/components/squid/development/squid-helm-generator.yaml
@@ -0,0 +1,23 @@
+apiVersion: builtin
+kind: HelmChartInflationGenerator
+metadata:
+  name: squid-helm
+name: squid-helm
+repo: oci://quay.io/konflux-ci/caching
+version: 0.1.275+b74f6fd
+valuesInline:
+  installCertManagerComponents: false
+  mirrord:
+    enabled: false
+  test:
+    enabled: false
+  cert-manager:
+    enabled: false
+  environment: release
+  resources:
+    requests:
+      cpu: 100m
+      memory: 128Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi
--- a/components/squid/staging/squid-helm-generator.yaml
+++ b/components/squid/staging/squid-helm-generator.yaml
@@ -0,0 +1,23 @@
+apiVersion: builtin
+kind: HelmChartInflationGenerator
+metadata:
+  name: squid-helm
+name: squid-helm
+repo: oci://quay.io/konflux-ci/caching
+version: 0.1.275+b74f6fd
+valuesInline:
+  installCertManagerComponents: false
+  mirrord:
+    enabled: false
+  test:
+    enabled: false
+  cert-manager:
+    enabled: false
+  environment: release
+  resources:
+    requests:
+      cpu: 100m
+      memory: 128Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi

Issue:

  1. Both components/squid/development/squid-helm-generator.yaml and components/squid/staging/squid-helm-generator.yaml are identical. If there's no difference between development and staging configurations, consider having a single base configuration and referencing it.
  2. The valuesInline.environment field is set to release in both development and staging configurations. This is likely incorrect for these environments.

Suggestion:

  1. In components/squid/development/squid-helm-generator.yaml, change environment: release to environment: development.
  2. In components/squid/staging/squid-helm-generator.yaml, change environment: release to environment: staging.
  3. If the resource limits or other settings are truly identical for development and staging, consider creating a components/squid/base directory with the common squid-helm-generator.yaml and have development and staging kustomizations reference it, possibly with patches for environment-specific values. However, for now, just updating the environment value is the most direct fix.

@avi-biton avi-biton force-pushed the squid-deploy branch 6 times, most recently from bc4136b to 5497c24 Compare September 17, 2025 13:42
@avi-biton
Copy link
Contributor Author

/retest

- Add squid applicationset
- Add squid to development and staging environments
- Exclude squid from production
- Add OWNERS file

Signed-off-by: Avi Biton <[email protected]>
Copy link

openshift-ci bot commented Sep 18, 2025

@avi-biton: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/konflux-e2e-v416-optional 40b2b44 link false /test konflux-e2e-v416-optional
ci/prow/appstudio-upgrade-tests 40b2b44 link false /test appstudio-upgrade-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@psturc
Copy link
Member

psturc commented Sep 19, 2025

/retest-required

Copy link
Contributor

@amisstea amisstea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Sep 19, 2025
Copy link

openshift-ci bot commented Sep 19, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisstea, avi-biton, psturc, yftacherzog

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit b1de394 into redhat-appstudio:main Sep 19, 2025
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants