-
Notifications
You must be signed in to change notification settings - Fork 316
(3.8.0 ‐ 3.15.1) User‐created Slurm maintenance reservations on static nodes cause an infinite kill relaunch loop
When a user creates a Slurm maintenance reservation on a static node (e.g. using scontrol create reservation), and that node subsequently needs replacement (due to a health check failure, prolog failure, etc.), ParallelCluster enters a tight kill/relaunch loop that prevents the node from recovering.
This occurs because ParallelCluster uses Slurm reservations internally to manage Capacity Block (CB) nodes. ParallelCluster names these reservations as pcluster-{capacity_block_reservation_id}. When an inactive Capacity Block is added to a cluster, the nodes are placed in a maintenance reservation and are not launched. When the CB becomes active, the nodes are taken out of the reservation and can be launched.
During the inactive period of the CB, nodes in a pcluster-{capacity_block_reservation_id} maintenance reservation are:
-
Excluded from the unhealthy node replacement list. ParallelCluster knows these nodes are intentionally not running and should not be replaced
-
Exempt from the replacement protection window. Because they are not expected to be launching, the 30-minute grace period for bootstrapping is skipped causing the nodes to be immediately terminated
When a customer creates their own Slurm maintenance reservation on a static node (e.g., scontrol create reservation ... Flags=maint), the node enters the same MAINTENANCE+RESERVED state. However, ParallelCluster does not recognize this reservation because it only tracks reservations it created (those prefixed with pcluster-). This causes the replacement protection window to be skipped, but the node is not excluded from the unhealthy node replacement list.
The result is that when a user places a maintenance reservation on a static node, clustermgtd launches a replacement instance successfully, the health check sees the node still in MAINTENANCE+RESERVED state, skips the replacement protection window, and terminates the replacement before it can finish bootstrapping. This loop repeats indefinitely until the reservation is removed.
The clustermgtd logs will show repeated loops of the node being terminated and replaced:
[slurm_plugin.slurm_resources:is_state_healthy] - WARNING - Node state check: node '{node-name}' in DOWN, node state: DOWN+CLOUD+MAINTENANCE+RESERVED+NOT_RESPONDING
[slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Found the following unhealthy static nodes: (x1) ['{node-name}']
[slurm_plugin.clustermgtd:_handle_unhealthy_static_nodes] - INFO - Setting unhealthy static nodes to DOWN
[slurm_plugin.clustermgtd:_handle_unhealthy_static_nodes] - INFO - Terminating instances backing unhealthy static nodes
[slurm_plugin.clustermgtd:_handle_unhealthy_static_nodes] - INFO - Launching new instances for unhealthy static nodes
The documented procedure to use Slurm reservations to resize clusters still applies when static nodes are healthy - in that case, they don't enter the kill/relaunch loop. Slurm reservations also work with dynamic nodes because the Slurm reservation prevents jobs from being scheduled on those nodes and, if there are no jobs allocated to those nodes, then they will not be relaunched.
ParallelCluster versions from 3.8.0 to 3.15.1 on all OSes using the Slurm scheduler.
Do not use Slurm maintenance reservations (scontrol create reservation ... Flags=maint) on static nodes. This conflicts with how ParallelCluster internally uses Slurm reservations for Capacity Block management. If a static node is already stuck in a replacement loop, remove the reservation (scontrol delete reservationname=<name>) and the next replacement will complete its normal bootstrap.