Skip to content

Syncer clears externally-injected scheduling gates on host Pods (e.g. kueue.x-k8s.io/admission) #4106

Description

@gola

What happened?

vCluster's syncer performs bidirectional spec alignment between virtual Pods (vPods) and host cluster Pods. When it detects that spec.schedulingGates differ between the vPod and the host Pod, it replaces the host Pod's gates with the vPod's gates.

This means that if an external controller (such as Kueue) injects a scheduling gate on the host Pod after creation — and the vPod does not carry that same gate — the syncer will strip the gate from the host Pod on the next sync cycle.

What did you expect to happen?

The syncer should only sync scheduling gates that it (or vCluster itself) manages. Externally-injected gates on the host Pod — such as kueue.x-k8s.io/admission — should be preserved and not overwritten by the vPod's gate state.

How can we reproduce it (as minimally and precisely as possible)?

  1. Deploy vCluster with a Kueue PodGroup workload.
  2. An external webhook injects kueue.x-k8s.io/admission scheduling gate on the host Pod (not on the vPod).
  3. The vCluster syncer detects that the vPod has no gates while the host Pod has one gate.
  4. The syncer patches the host Pod's spec.schedulingGates to null (or to match the vPod's empty list).
  5. The host Pod loses the kueue.x-k8s.io/admission gate.

Anything else we need to know?

Impact

Kueue PodGroup integration breaks:

Kueue uses scheduling gates (kueue.x-k8s.io/admission) to hold Pods in a SchedulingGated state while their Workload waits for quota admission. When the syncer clears this gate:

  1. The host Pod is no longer gated → Kueue's isGated() returns false
  2. Kueue's reconciler Step 7 (IsSuspended() → "nothing to do") is skipped
  3. The reconciler falls through to Step 8: !workload.IsAdmitted(wl)StopReasonNotAdmittedStop()Pod deletion

This causes premature deletion of queued Pods and disrupts the Kueue admission workflow.

Kubernetes scheduling gate semantics

Per the Kubernetes API documentation:

  • Scheduling gates can only be added at Pod creation time.
  • Gates can be removed after creation, but cannot be re-added.

This means once the syncer strips an externally-injected gate, it cannot be restored — the host Pod is permanently ungated.

Root cause (verified against latest main as of 2026-07-22)

In pkg/controllers/resources/pods/translate/diff.go, the calcSpecDiff() function does an unconditional replacement of host Pod gates with vPod gates:

// Latest main — unconditional overwrite (no diff check at all):
pObj.Spec.SchedulingGates = vObj.Spec.SchedulingGates

The previous version (v0.18.x / v0.30.0) had a conditional check via isPodSpecSchedulingGatesDiff(), but the latest code removed even that and directly assigns vPod gates to the host Pod. Either way, the result is the same: externally-injected gates on the host Pod are overwritten.

Community-accepted precedent in the same function: The Tolerations field is handled with a merge strategy that preserves host-only tolerations:

// Existing pattern in the same calcSpecDiff() — Tolerations merge:
newTolerations := append([]corev1.Toleration{}, vObj.Spec.Tolerations...)
for _, hostTol := range pObj.Spec.Tolerations {
    if !hasToleration(newTolerations, hostTol) {
        newTolerations = append(newTolerations, hostTol)
    }
}
pObj.Spec.Tolerations = newTolerations

This is exactly the pattern we propose for scheduling gates: merge vPod gates with host-only gates, preserving externally-injected ones.

Proposed solution (Option A)

Instead of replacing the host Pod's entire spec.schedulingGates with the vPod's, the syncer should:

  1. Preserve host Pod gates that are NOT present on the vPod (externally-injected gates).
  2. Sync gates that exist on both (update if needed).
  3. Remove host Pod gates that exist on the vPod but have been removed from the vPod (vCluster-managed gates being cleared).

This is a union-merge strategy: the host Pod retains hostGates ∪ vGates, minus any gates the vPod explicitly had before and has now removed.

Concretely, the diff logic should only remove gates from the host Pod that were previously synced from the vPod — not gates that were injected externally.

Environment

Host cluster Kubernetes version

Details
$ kubectl version
Client Version: v1.36.2
Kustomize Version: v5.8.1
Server Version: v1.34.2

vcluster version

Details
$ vcluster --version
v0.30.0

VCluster Config

Details
Not involve

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions