Node created by karpenter gets stuck in NodeNotReady state and recreates

[saitessell]

Description

Observed Behavior:

I am using karpenter version v0.30 with eks 1.30 and node created by karpenter momentarily goes to READY state and then to NODENOTREADY state with the schedule pods in pending phase for over 10min and node gets deleted automatically. A new node gets created and cycle continues for ever.

These are the node events

  Type     Reason                   Age                    From                   Message
  ----     ------                   ----                   ----                   -------
  Normal   Starting                 2m12s                  kube-proxy
  Normal   NodeHasSufficientPID     2m19s (x2 over 2m19s)  kubelet                Node ip-10-0-101-237.ap-south-1.compute.internal status is now: NodeHasSufficientPID
  Normal   Starting                 2m19s                  kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity      2m19s                  kubelet                invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  2m19s (x2 over 2m19s)  kubelet                Node ip-10-0-101-237.ap-south-1.compute.internal status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    2m19s (x2 over 2m19s)  kubelet                Node ip-10-0-101-237.ap-south-1.compute.internal status is now: NodeHasNoDiskPressure
  Normal   Synced                   2m19s                  cloud-node-controller  Node synced successfully
  Normal   NodeAllocatableEnforced  2m19s                  kubelet                Updated Node Allocatable limit across pods
  Normal   RegisteredNode           2m15s                  node-controller        Node ip-10-0-101-237.ap-south-1.compute.internal event: Registered Node ip-10-0-101-237.ap-south-1.compute.internal in Controller
  Normal   NodeReady                2m1s                   kubelet                Node ip-10-0-101-237.ap-south-1.compute.internal status is now: NodeReady
  Normal   DeprovisioningBlocked    115s                   karpenter              Cannot deprovision Node: Nominated for a pending pod
  Normal   Unconsolidatable         93s                    karpenter              Can't replace with a cheaper node
  Normal   NodeNotReady             35s                    node-controller        Node ip-10-0-101-237.ap-south-1.compute.internal status is now: NodeNotReady 

Expected Behavior:
Expected behaviour is to create nodes properly and schedule pods in them

Reproduction Steps (Please include YAML):

I am using the following provisioner :

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  annotations:
    karpenter.sh/provisioner-hash: '7636158900229219285'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"karpenter.sh/v1alpha5","kind":"Provisioner","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"tessell-k8s-tenant-samay-test-10-eks","helm.sh/chart":"karpenter-v0.30.0"},"name":"default"},"spec":{"consolidation":{"enabled":true},"kubeletConfiguration":{"kubeReserved":{"cpu":"100m","memory":"1500Mi"},"maxPods":50,"systemReserved":{"cpu":"80m"}},"providerRef":{"name":"default"},"requirements":[{"key":"node.kubernetes.io/instance-type","operator":"In","values":["t3a.xlarge"]},{"key":"kubernetes.io/os","operator":"In","values":["linux"]},{"key":"kubernetes.io/arch","operator":"In","values":["amd64"]},{"key":"karpenter.sh/capacity-type","operator":"In","values":["on-demand"]}]}}
  creationTimestamp: '2024-12-27T08:40:29Z'
  generation: 1
  labels:
    argocd.argoproj.io/instance: tessell-k8s-tenant-samay-test-10-eks
    helm.sh/chart: karpenter-v0.30.0
  name: default
  resourceVersion: '3301508'
  uid: ee8a6ec5-a956-4d66-aab3-6db1d1a052ba
  selfLink: /apis/karpenter.sh/v1alpha5/provisioners/default
status:
  resources:
    cpu: '4'
    ephemeral-storage: 41930732Ki
    memory: 16226708Ki
    pods: '50'
spec:
  consolidation:
    enabled: true
  kubeletConfiguration:
    kubeReserved:
      cpu: 100m
      memory: 1500Mi
    maxPods: 50
    systemReserved:
      cpu: 80m
  providerRef:
    name: default
  requirements:
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
        - t3a.xlarge
    - key: kubernetes.io/os
      operator: In
      values:
        - linux
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - on-demand

Versions:

• Chart Version: v0.30
• Kubernetes Version (kubectl version): 1.30

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jigisha620]

Hi @saitessell ,
With the release of Karpenter v1.0.0, the Karpenter team has dropped support for karpenter versions v0.32 and below. We recommend upgrading to the latest version of Karpenter and keeping Karpenter up-to-date for bug fixes and new features.