We can enable Machine Health Checks (MHC) for CAPI
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/healthchecking.html
The idea is that MHC constantly monitors node health. If a node stays broken longer than our configured timeout, MHC steps in, flags it as dead, and tells the cluster to delete the broken machine and spin up a brand new one to replace it. That's like a self-healing engine
And we can configure nodeStartupTimeoutSeconds , so e.g. when Machine is stuck in Provisioning, machine creation will be retried automatically
But I hit some bare-metal blocker
When I simulate a dead node (e.g. stop kubelet), after some time MHC marks machine for deletion as expected. CAPI tries to be polite and gracefully drain the node. But because the node is dead, Ceph loses a MON/OSDs and goes into HEALTH_WARN . And because of this warn state maintenance-controller does not allow to delete machine. So we have a deadlock here
I see some options here
- Just put aside this idea to introduce MHC and resolve all node issues manually
- We can adjust pre-deletion checks for machines, e.g. set smth like
nodeDrainTimeout to do a force drain, or configure maintenance-controller checks to allow machine deletion only if one node in not happy at one point in time
Maybe auto remediation in metal world would be too aggressive, maybe it would be cool, I am not sure
I'd love to hear your thoughts
We can enable Machine Health Checks (MHC) for CAPI
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/healthchecking.html
The idea is that MHC constantly monitors node health. If a node stays broken longer than our configured timeout, MHC steps in, flags it as dead, and tells the cluster to delete the broken machine and spin up a brand new one to replace it. That's like a self-healing engine
And we can configure
nodeStartupTimeoutSeconds, so e.g. when Machine is stuck inProvisioning, machine creation will be retried automaticallyBut I hit some bare-metal blocker
When I simulate a dead node (e.g. stop
kubelet), after some time MHC marks machine for deletion as expected. CAPI tries to be polite and gracefully drain the node. But because the node is dead,Cephloses a MON/OSDs and goes intoHEALTH_WARN. And because of this warn statemaintenance-controllerdoes not allow to delete machine. So we have a deadlock hereI see some options here
nodeDrainTimeoutto do a force drain, or configuremaintenance-controllerchecks to allow machine deletion only if one node in not happy at one point in timeMaybe auto remediation in metal world would be too aggressive, maybe it would be cool, I am not sure
I'd love to hear your thoughts