I'll work on this issue

/assign

/remove-help

What is current behaviour of the karpenter for nodeAffinityPolicy?

Currently Karpenter doesn't recognize either nodeAffinityPolicy, so it will always be treated as honor. Originally the linked PR was blocked on some 1.29 features and I haven't had the time to do the performance testing since we got unblocked on 1.29 testing. Those features have since been scoped out but the linked PR will add support for nodeAffinityPolicy, nodeTaintPolicy, and matchLabelKeys.

Hey everyone! Thanks for putting in the effort to support these new constraints! We just hit this exact issue in our clusters, where we are using matchLabelKeys together with topologySpreadConstraints to spread our pods across AZs.

I was wondering if you currently have a timeline for getting this PR merged? It seems the code is already ready for some time and it is just waiting for a rebase/performance test?

This has been backlogged for me for a little while, but I should have some bandwidth to get this wrapped out within the next week. Like you said, it really should only be performance testing and a rebase at this point that's left.

Amazing, thanks for the fast response. Really appreciate the work you are doing with karpenter!

We are also really awaiting for matchLabelKeys as now topologySpread is applied to all versions of a deployment which sometimes works not as optimal as we would like.

Hello everyone
In my company, we heavily rely on the matchLabelKeys setting and we've also encountered similar issues when new pods of the rollout were in the Pending because Karpenter didn't respect the matchLabelKeys and made the calculations taking into account pods of the previous revision. It caused an incident during the night preventing us from applying hotfixes to the production workloads.

Could you please share the ETA of this feature?

Hi, the lack of matchLabelKeys support, in particular, is also causing problems for us.

Is there anything we can do to help, test a custom build of Karpenter that supports this feature or something?

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

Hi, any update on when this feature may be available ?

Currently Karpenter doesn't recognize either nodeAffinityPolicy, so it will always be treated as honor.

I've observed different behavior, similar to what was described in this closed issue.

In my case, I have a zonal topologySpreadConstraint. There are nodes for all three zones (A, B, C) in the cluster, but only nodes in zones A and B (not C) match a label that's also set as a nodeSelector on a particular Deployment. This label is set by Karpenter by being included in NodePools, and those NodePools are associated exclusively with an EC2NodeClass that can only provision nodes in zones A and B.

The zonal topologySpreadConstraint has maxSkew=1, and whenUnsatisfiable=DoNotSchedule. Suppose I scale up this Deployment from 2 to 3, and assume replica 1 is in zone A and replica 2 is in zone B - the new replica should qualify for either zone A or zone B, and zone C should be disregarded because no existing or provisionable instances in zone C will match the required nodeSelector label. It appears as if kube-scheduler understands this, but Karpenter does not. If there is available capacity in A or B, kube-scheduler will properly schedule the pod. If there is no available capacity in A or B, Karpenter should provision a new instance, but instead it emits an error event no instance type satisfied resources ... and requirements ... topology.kubernetes.io/zone In [C] (no instance type met the scheduling requirements or had a required offering);, delaying the scheduling of the pod. This should not happen, since nothing indicated that C was a valid zone - including live instances matching the required nodeSelector (a form of affinity), or a NodePool/EC2NodeClass combination.

What's most confusing here is the difference in behavior between how kube-scheduler and Karpenter is applying the topologySpreadConstraint. aws/karpenter-provider-aws#3397 proposed adding an explicit zonal affinity rule as a workaround, but this should not be required since it's not required by kube-scheduler. Many users will not realize that the additional affinity rule is required (only for Karpenter), leading to surprise operational issues.