cfapi module components OOMKill under cluster resource load

[pPrecel]

---

Description

Multiple components deployed by the cfapi module show severe memory growth or OOMKills across all three tested resource types — pods, secrets, and configmaps.

Pods load test

pod (namespace/name)	baseline	500 res	1000 res	2000 res	3000 res	4000 res	5000 res	delta
korifi/korifi-controllers-controller-manager-5ddd86668-85tk6	45 Mi	166 Mi	286 Mi	724 Mi	804 Mi	1074 Mi	2363 Mi	+2318 Mi
kpack/kpack-controller-5c6866cf46-twcx6	62 Mi	163 Mi	325 Mi	538 Mi	699 Mi	1005 Mi	1023 Mi	+961 Mi

Secrets load test

pod (namespace/name)	baseline	500 res	1000 res	2000 res	3000 res	4000 res	5000 res	delta
korifi/korifi-controllers-controller-manager-5ddd86668-85tk6	965 Mi	991 Mi	1389 Mi	1560 Mi	2281 Mi	FAILING	FAILING	+1316 Mi
cfapi-system/cfapi-operator-989bb945-4mn9s	202 Mi	404 Mi	FAILING	FAILING	FAILING	FAILING	FAILING	+202 Mi
cfapi-system/contour-contour-57fd8d96b8-mn452	146 Mi	FAILING	FAILING	FAILING	FAILING	FAILING	FAILING	crash at 500

ConfigMaps load test

pod (namespace/name)	baseline	500 res	1000 res	2000 res	3000 res	4000 res	5000 res
cfapi-system/contour-contour-57fd8d96b8-mn452	80 Mi	75 Mi	82 Mi	86 Mi	FAILING	FAILING	FAILING

Three distinct components are affected:

1. korifi-controllers-controller-manager — the single largest memory consumer across all tested components. Grows ~52x under pods (+2318 Mi) and continues growing under secrets (+1316 Mi, OOMKills at 4000). Likely watches all pods and secrets cluster-wide.

2. cfapi-operator — stable under pod load, but OOMKills under secrets: doubles from 202 Mi to 404 Mi at just 500 secrets, then crashes. Likely watches secrets cluster-wide to detect registry credential changes.

3. contour — OOMKills under secrets at 500 resources (from 146 Mi baseline) and under configmaps at 3000 resources (from 80 Mi baseline). Likely watches configmaps and secrets cluster-wide for Ingress/HTTPProxy configuration.

Expected result

All components should handle large production clusters without OOMKilling. Watches should be scoped to namespaces and resource types directly managed by each component.

Actual result

• korifi-controllers-controller-manager grows ~52x under pods and OOMKills under secrets at 4000 resources
• cfapi-operator OOMKills at 500 secrets
• contour OOMKills at 500 secrets and at 3000 configmaps
• kpack-controller grows ~17x under pod load

Steps to reproduce

1. Deploy Kyma with the cfapi module enabled
2. Create 5000 fake pods / secrets / configmaps using KWOK (one resource type per test run)
3. Observe affected components OOMKilling

Possible Solutions

From a business perspective, both controllers only need to be aware of pods they directly manage (CF apps / kpack builds). Watching all pods in the cluster is unnecessary and causes unbounded memory growth.

1. Scope the pod cache to CF-managed namespaces and workloads

korifi manages CF apps in dedicated org/space namespaces. The controller cache can be restricted to those namespaces only, preventing the accumulation of unrelated pod state:

cache.Options{
    DefaultNamespaces: map[string]cache.Config{
        "korifi":       {},
        "kpack":        {},
        // add CF org/space namespaces as needed
    },
}

Alternatively, CF-managed pods already carry predictable labels (e.g. korifi.cloudfoundry.org/app-guid). Those labels can be used as informer selectors to further bound the cache:

cache.Options{
    ByObject: map[client.Object]cache.ByObject{
        &corev1.Pod{}: {
            Label: labels.SelectorFromSet(labels.Set{"korifi.cloudfoundry.org/app-guid": ""}),
        },
    },
}

2. Use client.Reader (uncached) for read-only lookups

For operations that only need to look up a resource once (not watch it continuously), use the API reader that bypasses the cache:

type MyReconciler struct {
    client.Client
    APIReader client.Reader // set to mgr.GetAPIReader()
}

// Use APIReader for one-off lookups instead of r.Client.Get(...)
err := r.APIReader.Get(ctx, key, obj)

The APIReader and cache exclusion should be used together: APIReader for the call site, and cache configuration to prevent the informer from being populated by other code paths.

Anything else?

• Load test results: https://github.tools.sap/kyma/warden/issues/173#issuecomment-24684347
• Test script: Load testing toolset manager-toolkit#66
• controller-runtime cache options: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/cache#Options

[pPrecel]

Additional solution: strip unused fields from cached objects using a cache transform

A DefaultTransform function strips fields that controllers never read before storing objects in the cache. For korifi-controllers-controller-manager and kpack-controller, which both show the worst pod-load growth, removing pod containers, volumes, env vars, and annotations can cut per-pod memory cost by ~90%. KEDA applied this approach in kedacore/keda#7748:

import (
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func CacheObjectTransform(obj any) (any, error) {
    if accessor, ok := obj.(metav1.Object); ok {
        accessor.SetManagedFields(nil) // safe for all types
    }
    if pod, ok := obj.(*corev1.Pod); ok {
        // keep only what the controller actually reads
        nodeName := pod.Spec.NodeName
        pod.Spec = corev1.PodSpec{NodeName: nodeName}
        pod.Annotations = nil
        phase := pod.Status.Phase
        conditions := pod.Status.Conditions
        pod.Status = corev1.PodStatus{Phase: phase, Conditions: conditions}
    }
    return obj, nil
}

Register it in manager options:

ctrl.Options{
    Cache: cache.Options{
        DefaultTransform: CacheObjectTransform,
    },
}

Adapt the retained fields per component. For cfapi-operator and contour (secrets/configmaps issue), the same transform strips ManagedFields from those resource types, reducing per-object overhead.