[Feature] Ray Autoscaler integration with Kubernetes gang scheduling and workload lifecycle management

[OneSizeFitsQuorum]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Motivation

KubeRay already supports integration with Kubernetes batch schedulers such as Volcano for gang scheduling.

However, as Ray workloads become more dynamic and elastic, there are still open questions around how Ray Autoscaler should interact with external schedulers when both components influence workload lifecycle.

Today we have multiple control loops:

• Ray Autoscaler

  • manages Ray worker group scaling
  • decides when to add/remove resources based on Ray workload demand

• Kubernetes scheduler / Volcano

  • manages pod placement
  • performs gang scheduling
  • handles preemption and resource reclamation

These components may make independent decisions about the same workload, which can lead to unexpected behaviors during dynamic resource changes.

---

Example Scenario

Consider a Ray cluster running on a heterogeneous Kubernetes cluster:

• CPU-only nodes
• GPU nodes

Initially, due to resource availability, some CPU Ray worker pods may be scheduled onto GPU nodes.

Later, a GPU workload arrives and requires those GPU nodes.

Ideally:

1. The scheduler identifies that GPU workloads have higher priority.

2. CPU worker pods are preempted from GPU nodes.

3. CPU workers are rescheduled onto CPU nodes.

4. Ray Autoscaler adjusts node groups accordingly:

   • scale down unused GPU capacity
   • scale up CPU capacity if needed

However, during this transition:

• Ray workers may temporarily disappear.
• Ray Autoscaler may interpret missing workers as insufficient capacity and scale out.
• The scheduler may keep the workload waiting due to gang constraints.
• The workload PodGroup may enter an InQueue state.

This creates potential conflicts between scheduler decisions and Ray Autoscaler decisions.

---

Open Questions

1. How should Ray Autoscaler react to scheduler-driven pod lifecycle changes?

When a Ray worker pod disappears because of scheduler actions:

• preemption
• eviction
• gang scheduling transition

Should Ray Autoscaler:

• immediately treat it as a missing worker and create replacement capacity?
• wait for the scheduler to complete rescheduling?
• distinguish between workload failure and scheduler-driven migration?

Is there a recommended signal/mechanism for Ray Autoscaler to understand the reason behind worker changes?

---

2. Should Ray Autoscaler be aware of gang scheduling states?

For workloads requiring a minimum number of workers:

Example:

desired workers: 10
running workers: 6
PodGroup: InQueue

Currently, Ray Autoscaler may see missing workers and attempt scaling actions.

Should Ray Autoscaler:

• understand external scheduler admission states?
• avoid scaling while the workload is waiting for gang admission?
• coordinate with scheduler-provided signals such as PodGroup status?

---

3. What is the recommended ownership boundary between KubeRay and external schedulers?

Possible models:

Scheduler-driven model

Volcano/Kueue owns:

• placement
• admission
• preemption

Ray Autoscaler only reacts to final cluster state.

Questions:

• How does Ray avoid overreacting to temporary scheduler states?

Ray-aware model

Ray Autoscaler understands:

• gang requirements
• scheduler constraints
• migration events

Questions:

• Should KubeRay expose additional integration points?
• Should Ray workload requirements be propagated to Kubernetes scheduling frameworks?

---

Goal

The goal is to define a better integration model between:

• KubeRay / Ray Autoscaler
• Kubernetes gang schedulers (Volcano, Kueue, etc.)
• Kubernetes cluster autoscaling

for elastic Ray workloads.

This is not limited to distributed training workloads. The same challenges may appear in:

• Ray placement groups
• actor workloads with resource dependencies
• mixed CPU/GPU clusters
• dynamic workload rebalancing

We would appreciate feedback from the community on the recommended architecture and potential improvements.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[OneSizeFitsQuorum]

issue in volcano volcano-sh/volcano#5650

[machichima]

@OneSizeFitsQuorum Ray Autoscaler just expand pods rather than nodes. So I think this should be handle by k8s node scheduler instead? Maybe user can set preferredDuringSchedulingIgnoredDuringExecution?

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity

[win5923]

Hi @OneSizeFitsQuorum, would Priority-Aware Worker Group Selection address your use case?
ray-project/ray#62996

My understanding is that this is primarily a Kubernetes scheduling concern. CPU pods can be kept off GPU nodes using mechanisms such as nodeSelector, nodeAffinity, and tolerations, so KubeRay may not need to handle this explicitly.

[OneSizeFitsQuorum]

@win5923 @machichima Thanks for the pointers. I went through the code on both sides and I agree your direction (nodeAffinity/tolerations, KubeRay doesn't handle this explicitly) is viable for the basic scenario — with one prerequisite that's easy to miss.

What works today (config-only, no KubeRay/Volcano code): two worker groups — a baseline CPU group (MinReplicas=MaxReplicas=N, requiredDuringScheduling CPU nodeAffinity, no GPU toleration, volcano.sh/preemptable: "false") and an overflow/downgrade group (MinReplicas=0, MaxReplicas=M, preferredDuringScheduling CPU + GPU toleration). Because the overflow group's MinReplicas=0 contributes 0 to the PodGroup MinMember (CalculateMinReplicas, util.go:420), the gang floor = baseline + head; overflow pods are always surplus above the floor, so Volcano can preempt them freely when a higher-priority GPU job needs the GPU node, without breaching the floor (gang.go:113, priority-independent). KubeRay then recreates the preempted overflow worker from the overflow template (preferred CPU) → it lands on CPU when CPU has room.

(Why two groups instead of one group with min/max: the floor math is actually the same, but splitting lets us hard-pin the floor-protected baseline pods to CPU with no GPU toleration, so the pods occupying GPU nodes are always evictable surplus, never floor-protected pods that can't be moved — which is exactly the original problem this issue describes.)

Prerequisite: enableInTreeAutoscaling must be ON. calculatePodGroupParams (volcano_scheduler.go:200) only uses MinReplicas+1 as the floor when autoscaling is on; with autoscaling off it uses DesiredReplicas+1 = running count, so the floor is always saturated and nothing is preemptable. This is the one gotcha that silently breaks the whole setup.

Scope, so future readers aren't misled:

• preferredDuringScheduling is best-effort (nodeorder nodeAffinity weight, default 2). If CPU is busy, a recreated overflow pod may reland on GPU and get preempted again — self-correcting but thrashy; a hard pin would prevent the GPU downgrade in the first place.
• Evicted workers' in-flight Ray tasks/actors die (Volcano preempts via hard Pod Delete, no grace); Ray retries them — acceptable for fault-tolerant workloads. True graceful drain is separate (Ray's GCS DrainNode RPC already ships; KubeRay wiring it before removal, along the lines of [ray-operator] Add graceful Pod termination via Ray's DrainNode/DrainRaylet RPC #5005, is future work).
• Overflow groups must use NumOfHosts=1 (reconcileMultiHostWorkerGroup deletes the whole replica group on one missing host).
• Case (b) both-full: the overflow pod goes Pending and relies on cluster node autoscaling to add a CPU node — but JesseStutler noted on #5650 that existing node autoscalers simulate via kube-scheduler and can't satisfy real gang, so this case is genuinely open and tracked there.

On ray#62996 (Priority-Aware Worker Group Selection): complementary, not required — it steers the autoscaler's scale-UP toward the CPU group, while the nodeAffinity/preemptable config above handles placement + eviction. They compose, but the basic scenario doesn't depend on it.

If thrash/over-reaction turns out material, the minimal KubeRay-side follow-up would be making the recreate eviction-aware (read the DisruptionTarget condition Volcano sets on the victim, back off recreate for preempted surplus pods) — safe because those pods are surplus — but I'd only add that if testing shows it's needed.

For transparency: the above is my own reading of both codebases, may not be fully correct, and I'm sharing it early in case others with the same need want to try it; I'll run real tests and report back with results.

[andrewsykim]

cc @yaroslava-serdiuk @gangli113

[Smallfu666]

A datapoint from the HPC side that maps onto Questions 1–3: we run Ray (2.56) bare-metal on Slurm across multiple nodes, which is effectively the extreme end of the "scheduler-driven model". It provides a useful contrast: Slurm grants the full allocation before any Ray process starts (gang admission is atomic), and we run Ray within that fixed resource set without an active autoscaling loop — so Ray never observes partially admitted capacity.

Two observations from that deployment that seem relevant here:

• On Q2: the failure mode described (the autoscaler reacting to missing workers while the PodGroup is InQueue) is exactly the partially-admitted state that atomic admission avoids. Rather than making the Ray autoscaler infer admission from pod counts, KubeRay — or its Kubernetes-facing node-provider layer — could defer worker reconciliation while the corresponding workload has not reached an admitted state (PodGroup phase, or Kueue's Workload Admitted condition), without requiring the Ray Autoscaler itself to understand Volcano- or Kueue-specific APIs. That reproduces the property that makes the HPC model stable: don't react until admission has settled.

• On Q1: Ray-side node identity won't help here — a restarted/rejoining worker registers as a new raylet with a new node ID, even on the same machine. Without an external drain or scheduler-originated termination signal, Ray just sees the old raylet disappear and a newly identified raylet join; it cannot infer that this pair of events is a scheduler-initiated replacement rather than an unrelated failure-and-join. (We also hit an integration pitfall where our Slurm-side bookkeeping keyed workers by HOSTNAME while Ray registered the raylet under its selected node address and generated node ID, so hostname-based correlation was unreliable until we normalized identity explicitly.) So the "why did this worker disappear" signal realistically has to come from the Kubernetes side — e.g. the DisruptionTarget condition @OneSizeFitsQuorum mentioned — with KubeRay translating it, which supports keeping the eviction-aware backoff on the KubeRay side rather than making the Ray autoscaler scheduler-aware.

FWIW the two-worker-group recipe above looks right to me. I can run a scoped kind + Volcano reproduction of that two-worker-group scenario and report the observed admission and recreation behavior, if that datapoint would be useful.

[yaroslava-serdiuk]

@OneSizeFitsQuorum Could you clarify the exact behavior you need for autoscaled Ray workloads?

Specifically, are you looking to:

1. Deferred Gang Scale-Up: Hold off on launching any scale-up pods until the entire target capacity can be provisioned together?
2. Reserved capacity for scale-up: The autoscaled workers must run at full requested capacity immediately upon scaling up?

[Smallfu666]

Following up on my earlier comment: I ran the scoped kind + Volcano reproduction of @OneSizeFitsQuorum's two-worker-group recipe. Setup, then observations. Everything below is from one scenario on one small box, so please read it as a datapoint, not a general claim.

Setup. kind v0.32.0 with kindest/node:v1.29.0 (K8s v1.29.0), 3 nodes on a single 12-CPU host: control-plane + one worker labeled node-type=cpu + one worker labeled node-type=gpu with taint nvidia.com/gpu=present:NoSchedule (no real GPUs — taint/affinity semantics only, and both kind "nodes" share the same physical machine). Volcano v1.15.0 (release installer YAML, default queue), KubeRay operator v1.6.2 (--batch-scheduler=volcano), Ray 2.52.0 (autoscaler v2 head sidecar). RayCluster per the recipe: head (1 CPU, num-cpus=0, pinned to the cpu node); cpu-baseline group replicas=minReplicas=maxReplicas=2, 1 CPU each, requiredDuringScheduling node-type=cpu, no GPU toleration, volcano.sh/preemptable: "false"; overflow group minReplicas=0, maxReplicas=3, preferredDuringScheduling cpu (weight 100) + GPU toleration; enableInTreeAutoscaling: true. Requests are scaled down to fit the box, which shifts absolute numbers but not the admission semantics being tested.

1) Admission order. The floor math in the recipe is what actually materializes: the PodGroup was created with minMember: 3 (head + 2 baseline; overflow's minReplicas=0 contributed 0). On apply, the operator created all pods in one reconcile pass, and the PodGroup went Pending → Inqueue → Running within ~2s; all pods (including the surplus overflow pod, since capacity allowed) were bound in the same scheduling cycle — the gang event reads 0/4 tasks in gang unschedulable: pod group is ready, 3 minAvailable, 4 Binding. So admission was atomic in the positive direction.

The negative direction also held. A variant cluster whose baseline requests made the floor infeasible (2×8 CPU, minResources: cpu=17 > any node) got all its pods created once by the operator, then sat with the PodGroup stuck in Inqueue and zero pods bound — the gang event explicitly shows 4/4 tasks in gang unschedulable: ... 2 Schedulable, 2 Unschedulable, i.e. the two pods that individually fit were still not started. No partial start, and no operator recreate churn while stuck (each pod created exactly once over the observation window). One consequence worth noting for Q2: since the Ray autoscaler runs as a head-pod sidecar, it never starts in this never-admitted state — pre-admission there is no Ray-side control loop to conflict with the scheduler; the conflict surface only exists post-admission.

Two small details observed in passing: (a) with enableInTreeAutoscaling: true the minMember/minResources we saw match the CalculateMinReplicas+1 path the recipe describes (we did not run the autoscaling-off variant); (b) minResources is computed from the CR templates, so the injected autoscaler sidecar's requests (default 500m/512Mi) are not included — actual head-pod requests slightly exceed the PodGroup's minResources share (3.5 CPU scheduled vs cpu: 3 in minResources here). Irrelevant at our scale, but under tight quota a PodGroup could pass enqueue while its real pods need a bit more.

2) Behavior under resource shortage (the "missing workers while gang-blocked" case).
Two sub-cases:

Floor breached post-admission (cordoned the cpu node, then deleted one baseline worker): the operator created exactly one replacement pod, which stayed Pending (0/3 nodes are unavailable: 1 node(s) were unschedulable, 2 node(s) didn't match Pod's node affinity/selector); the PodGroup dropped to phase=Unknown, running=2 < minMember=3 with recurring gang-unschedulable warnings. Over the ~4.5-minute breach window the autoscaler took no action (its log shows only idle-tick lines protecting the remaining baseline worker via min_worker_nodes) and there was no delete/recreate churn from either component. On uncordon, the same pending pod was bound in place and the PodGroup returned to Running within ~13s. Caveat: there was no Ray task demand during this window, so this only shows the no-demand behavior. Side note: RayCluster.status.state remained ready throughout the breach.

Demand-driven scale-up beyond capacity (closest to the issue's "desired 10 / running 6 / InQueue" shape): with the overflow group at 8 CPU/pod, a driver submitted 3 × num_cpus=8 tasks. The autoscaler scaled overflow 0→3 in one shot; one pod bound to the cpu node, one landed on the gpu node (preferred-CPU lost under pressure, toleration allowed it — exactly the recipe's best-effort caveat), and the third stayed Pending (1 node(s) had untolerated taint(s), 2 Insufficient cpu). The PodGroup stayed Running and minMember stayed 3 throughout — overflow expansion never touched the gang floor, as predicted. The one over-reaction we did observe is here: the autoscaler v2 instance reconciler has a 300s timeout for instances stuck in ALLOCATED (pod created, Ray node never started):

Update instance ALLOCATED->TERMINATING (... type=overflow,
  cloud_instance_id=repro5002-overflow-worker-tbmts, ray_id=): timeout=300s at status ALLOCATED
Terminating worker pods: ['repro5002-overflow-worker-tbmts']
Adding 1 node(s) of type overflow.        <- same second, demand still unmet

i.e. while demand persists and capacity never arrives, the pending pod is deleted and recreated on a ~5-minute period. We observed one full cycle (the second was cut short because the driver exited; scale-down after that was clean — running overflow workers were drained after the 60s idle timeout and the pending pod was removed). This is a concrete instance of Q1/Q2's theme: the pod-level reconciler acts on "Ray node didn't start within timeout" without knowing the scheduler already has a definitive answer ("does not fit"), producing API/scheduler churn with no state change. It is also precisely the kind of loop that an admission-aware backoff (defer the timeout while the pod's group/pod is scheduler-pending for capacity reasons, or read the scheduling condition) would suppress. FWIW this datapoint seems relevant to the deferred-gang-scale-up option @yaroslava-serdiuk asked about above: the churn here comes precisely from scale-up pods being launched (and timed out) before the capacity to run them exists.

3) Recreation behavior. Deleting a running baseline worker on a healthy cluster: the operator recreated it ~1s later, and the replacement joined the existing PodGroup rather than triggering a full re-admission — Volcano's gang check flags one cycle (pod group is not ready, 1 Releasing, 2 Running, 3 minAvailable), then binds on the next (pod group is ready, 1 Binding); delete→bound was ~1-2s, with the PodGroup phase flapping Running → Unknown → Pending → Running in ~2s. The autoscaler observed passively (instance RAY_RUNNING->TERMINATED, new instance ALLOCATED) and took no scaling action. One detail consistent with what I described earlier from the Slurm side: the replacement raylet registered with a new Ray node ID on the same template — Ray-side identity does not survive pod recreation, so any "this disappearance is a scheduler-initiated replacement" signal indeed has to come from the Kubernetes side.

Also observed, for completeness: with minReplicas=0, the initial overflow worker was idle-terminated by the autoscaler after the default 60s (graceful: DrainNode RPC succeeded before pod delete), so overflow workers only exist under demand — consistent with the recipe's intent, just worth knowing when you deploy it and the overflow pod "disappears."

vs. the recipe's predictions: floor composition (minMember = head + fixed groups only), overflow-as-evictable-surplus never affecting the floor, recreation from the overflow/baseline template, and the "preferred CPU may reland on GPU under pressure" caveat all matched observation. What this test did not exercise: actual Volcano preemption by a higher-priority job (so no data on preemptable: "false" honoring or preemption-triggered thrash), the enableInTreeAutoscaling-off failure mode, NumOfHosts>1, Kueue, real GPU resources, or cluster node autoscaling. All timeouts were defaults (60s idle, 300s ALLOCATED).

Raw manifests/logs are available if anyone wants them. Again: one scenario on one 12-CPU box with simulated node topology — the 300s-churn observation in particular is specific to Ray 2.52.0's autoscaler v2 + KubeRay v1.6.2 defaults and should be re-checked at real scale before drawing conclusions from it.

[OneSizeFitsQuorum]

@Smallfu666 @yaroslava-serdiuk Thank you for the detailed reproduction and for helping clarify the expected behavior. The explicit environment, positive and negative admission cases, lifecycle transitions, logs, and stated limitations make this a very useful datapoint. In particular, it provides preliminary independent validation of most of the assumptions behind the two-worker-group configuration: the gang floor was calculated from the head and baseline workers, the overflow group with minReplicas=0 remained outside minMember, admission was atomic when the floor was either feasible or infeasible, replacement Pods were recreated from the expected worker-group templates, and preferredDuringScheduling behaved as a best-effort preference under resource pressure.

The next useful validation would be the part that has not yet been exercised: introducing a higher-priority GPU workload and allowing Volcano to actually preempt an overflow CPU worker running on the GPU node. It would be useful to confirm that the baseline workers remain protected by the gang floor and preemptable: "false", only the overflow worker is selected as the victim, KubeRay creates exactly one replacement from the overflow template, and the replacement prefers a CPU node when capacity becomes available. This would complete the end-to-end validation of the original migration scenario and show whether preemption introduces any additional recreation churn.

To answer the two questions directly:

1. Deferred Gang Scale-Up: This could be a useful optimization for workloads that require the requested scale-up capacity to become available atomically, but I do not think it is a strict requirement for the basic elastic overflow scenario or something that necessarily needs to be the default. The initial gang floor is already admitted atomically, while overflow workers remain outside minMember. Holding all overflow workers until the complete target capacity is available could prevent partial resource consumption for some workloads, but it could also delay useful partial capacity. An opt-in policy may therefore be more appropriate.

2. Reserved capacity for scale-up: No, I do not currently expect all autoscaled workers to run at the full requested capacity immediately, nor do I expect the entire capacity to be reserved in advance. It should be acceptable for workers to start incrementally and for some Pods to remain Pending while Kubernetes or the cluster autoscaler obtains capacity. The main expectation is predictable convergence rather than immediate fulfillment of the complete scale-up request.

Regarding the observed 300-second ALLOCATED timeout, it may be possible to distinguish between a Pod that is valid but temporarily waiting for schedulable capacity and an instance that is genuinely stuck or has encountered a retryable launch failure. In the former case, deleting and recreating the Pod is unlikely to change the outcome; in the latter case, replacement may be useful. However, recreating a Pending Pod once every five minutes may not represent significant overhead in practice. Keeping the current fixed-timeout behavior without introducing scheduler-state classification is also a reasonable and relatively simple engineering tradeoff. Whether this scenario warrants special handling likely depends on how frequently it occurs and whether the resulting API and scheduling churn has measurable operational impact. I would be interested in the maintainers' view on whether distinguishing "waiting for capacity" from a genuinely stuck instance would justify the additional complexity.

Overall, this reproduction provides preliminary validation that the proposed two-worker-group configuration is viable. Actual Volcano preemption by a higher-priority GPU workload still needs to be exercised, but most of the core mechanics behaved as expected. The original use case therefore appears largely addressable through the existing configuration rather than requiring a new cross-component architecture. The remaining questions seem to be focused refinements and engineering tradeoffs, which we can revisit based on further testing and community feedback.