Request for Backport of Bug Fix for EBS CSI Driver Taint Issue to Karpenter v0.37.X

[prad9192]

Description

We have encountered a bug related to the EBS CSI Driver taint being incorrectly added back by Karpenter after the driver has already removed the taint.

• Current Behavior: The taint from the EBS CSI Driver is being added back by Karpenter, causing unexpected behavior.
• Expected Behavior: Karpenter should not add back the taint once the driver removes it.

This bug has already been addressed and fixed in the following PRs:

• Original Fix: kubernetes-sigs/karpenter#1658
• Cherry-Pick: kubernetes-sigs/karpenter#1705

However, this fix was included starting from Karpenter version v1.0.4.

---

Impact

We are currently running Karpenter version v0.37.6 and have started encountering this issue in one of our clusters.

While we are in the process of upgrading to version v1 following the upgrade guide, which recommends upgrading to the latest patch within the same minor version, the issue persists in v0.37.x.

Issue details - aws/containers-roadmap#2470

---

Request

We kindly request a backport of the aforementioned patch to the v0.37.x release series.

This would ensure that users following the upgrade process can avoid encountering this issue and experience a smoother transition to version v1.

---

Additional Information

• Karpenter Version: v0.37.6

Thank you for considering this request. Let us know if further details or testing assistance is required.

[njtran]

If you're already upgrading to v1 and you see it's been backported to v1.0.x, is there a reason you can't upgrade to that version right now?

[prad9192]

That's a good point. We have a custom controller that serves as an interface between developers and Karpenter resources.

At the moment, the controller's Go module is built on Karpenter version 0.35.5, so it will be interesting to test its compatibility with v1. Currently, it works with Karpenter version 0.37.6.

It's a bit more complicated than simply upgrading from 0.35.2 to 0.37.6 and then to v1. This is because we're managing Karpenter deployments using Helm templates by leveraging the manifests within cluster resource sets.

We are currently handling the upgrade in three stages:

1. Upgrade Karpenter to version 0.37.6.
2. Update the custom controller to reference Karpenter's v1 module.
3. Once the above changes are rolled out across all tiers, proceed with upgrading Karpenter to v1.

For more details, I have an ongoing discussion here for reference: GitHub Discussion #1870.

Regarding your original query, I'll get back to you after conducting some tests in the lower tiers.

[jonathan-innis]

/triage needs-information

[prad9192]

Hey Folks,

We upgraded Karpenter to v1.0.8, I can confirm that this issues does not exist with v1.0.8