[Umbrella] Autoscaler improvements

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

TODO:

• Define the scope of Autoscaler V2 beta
• List the important issues need to solve (V1 & V2).

This umbrella issue covers two topics:

• Autoscaler V2 towards beta
• Autoscaler stability improvements (V1 + V2)

Reliability

Top priority:

• Autoscaler should not terminate worker Pods with running Actor / Task
• Autoscaler should not crash because of CR spec
• A Job should be able to finish.

• [Core] Fix token expiration for ray autoscaler ray#48481
• [Ray Core] The node storing the actor will be kill unexpectedly when autoscaler is turned on ray#46172
• [Bug] autoscaler crash when resources.limt not set #2612
• [Autoscaler] Issue with RayCluster autoscaler and hanging non existent tasks ray#48950
• [core][autoscaler] Autoscaler V2 - cluster state version number  ray#35873 (GCS FT)
• [core][autoscaler][v1] Autoscaler overprovisions nodes when strict placement group is rescheduling ray#40212
• [Bug] Autoscaler sideacr crashes, bringing down head pod, if request exceeds max pod replicas #2385
• [core][autoscaler] Autoscaler doesn't scale up correctly when the KubeRay RayCluster is not in the goal state ray#48909
• [core][autoscaler] Fuse scaling requests together to avoid overloading the Kubernetes API server ray#49150
• [core][autoscaler] Set timeout for requests sent to K8s API server ray#48513
• [core][autoscaler] Fix incorrectly terminating nodes misclassified as idle in autoscaler v1 ray#48519
• [core][autoscaler] Autoscaler v2 does not honor minReplicas/replicas count of the worker nodes and constantly terminates after idletimeout  ray#48623
• [Core]: Fix ConnectionError on Autoscaler CR lookups in K8s clusters with custom DNS for Kubernetes API. ray#48541
• [Autoscaler] Autoscaler gets into infinite cycle of removing and adding nodes, never satisfies placement group ray#50783
• [core][autoscaler][v2] do not removing nodes for upcoming resource requests ray#51321
• [Cluster] Autoscaler frequently fails to scale down workers ray#51585
• [Autoscaler][V2] Updating max replicas while Pods are pending causes v2 autoscaler to hang ray#50868
• [Autoscaler][V2] Autoscaler V2 does not honor 'Conservative' upscaling mode ray#50259
• [Core] Ray Worker stuck in launching state - Azure AKS ray#45775
• [Ray Autoscaler] Ray autoscaler does not scale up effectively and fast  ray#45373
• [Ray Core] --no-monitor flag causes ray cluster status to break, causing the Ray Dashboard to be invisible ray#40911
• [Autoscaler] starts extra workers than necessary  ray#36926
• [Autoscaler] autoscaler on kuberay deletes pod with workloads  ray#35560
• [Autoscaler][V2] Autoscaler fails to delete idle KubeRay Pod ray#52264
• [core][autoscaler] let AWS node provider retry instance retrieval to handle eventual consistency ray#52200

Usability

• [core][autoscaler] Make some Autoscaler configurations configurable ray#49200
• [Core][Autoscaler] Configure idleTimeoutSeconds per node type ray#48813
• [Core][Autoscaler] API for getting max cluster resources ray#49501
• [autoscaler] dynamically set min_nodes during interactive sessions ray#47248
• [core][Autoscaler] : Autoscaler is unable to scale nodes for a placement group  ray#36836

Testing

• [Umbrella] Ray Autoscaling tests #2173

Observability / Debuggability

• [core][autoscaler] Health check logs are not visible in the autoscaler container's stdout ray#48905

• De-noise autoscaler logs. Currently, the autoscaler loops with Fetched pod data, outputting the state of the RayCluster even with no changes to the resources requested or allocated. This can make it fairly difficult to debug autoscaler logs. It'd be useful to provide the option to output relevant logs only on Autoscaler updates. (issue to be created)

  • Not urgent. We can set up env var to configure it.

• [core][autoscaler] Better observability for request resources  ray#37959

• [autoscaler] Refactor ray status output code ray#37856 (@ryanaoleary)

• [core][autoscaler] Add Pod names to the output of ray status -v ray#51192

Refactor

• [core][autoscaler] Remove unused deprecated flags for AutoscalerMonitor ray#48919
• [core][autoscaler] Make the head group name consistent with the head Pod's ray.io/group label. ray#48840
• [core][autoscaler] Remove local dependencies on Ray for KubeRay autoscaling e2e tests ray#48566
• [core][autoscaler] Remove worker_rpc_drain from Autoscaler V2 ray#49581
• [core][autoscaler] Remove provider_exists and allow_multiple ray#49812
• Update doc for v2 if v2 is used by default

Backlogs

• [Feature] Don't submit scale requests if the worker group is suspended #2666 (relies on KubeRay v1.3)
• [autoscaler][core][v2] Support all clouds for instance type name from config  ray#39735
  • install_ray is only used when disable_node_updaters is false, so KubeRay doesn't use this.
• [Core] Fractional unit instance resource may not trigger autoscaling ray#39987
• [autoscaler] Able to inspect autoscaler state/cluster config ray#37838

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

/assign @ryanaoleary

[ryanaoleary]

TODO: I'll leave a comment outlining the V2 beta scope and remaining issues to solve for V1 & V2.

[kevin85421]

@ryanaoleary thanks! You can compile a list of issues, and we can schedule a meeting to go through them one by one.

[ryanaoleary]

The issues that I think we ought to complete before considering Autoscaler v2 in Beta can be broken down into observability improvements and reliability bug-fixes.

Observability:

• Provide an API to persist worker logs after autoscaler scale-down, this is covered in the export API proposal (this may fall outside the scope of KubeRay v1.3)
• De-noise autoscaler logs. Currently, the autoscaler loops with Fetched pod data, outputting the state of the RayCluster even with no changes to the resources requested or allocated. This can make it fairly difficult to debug autoscaler logs. It'd be useful to provide the option to output relevant logs only on Autoscaler updates. (issue to be created)
• Direct formatting to output ray status: [autoscaler] Refactor ray status output code ray#37856
• Refactor request resources in autoscaler logs: [core][autoscaler] Better observability for request resources  ray#37959
• Generate node IDs: [autoscaler] Autoscaler should avoid using ip address to identify nodes (as node id of node provider) ray#19086
• Expose API to expose autoscaler state: [autoscaler] Able to inspect autoscaler state/cluster config ray#37838

Reliability:

• Maintain correctness of cluster state version: [core][autoscaler] Autoscaler V2 - cluster state version number  ray#35873
• Reserve custom accelerators: [core][autoscaler][v2] Reserve nodes with accelerator nodes.  ray#43079
• Further v2 autoscaler tests in Ray Core and e2e v2 Autoscaler tests using the latest Ray image in KubeRay.
• [autoscaler][core][v2] Support all clouds for instance type name from config  ray#39735

Several of the completed issues mentioned in the issue description fix the main reliability issues found within the v1 Autoscaler, and from my manual testing with multiple CPU and GPU worker-groups I've seen consistent behavior. Additionally, new features in the v2 autoscaler such as configuring idle node timeouts by node type will enable users with more fine-grained control of their workloads and minimize the amount of autoscaling errors we were previously seeing. We should also consider it a requirement to ensure reliable testing for CPUs, GPUs, and custom accelerators before considering v2 beta.

[kevin85421]

TODO: confirm the following two issues.

• [autoscaler] Autoscaler should avoid using ip address to identify nodes (as node id of node provider) ray#19086
• [core][autoscaler][v2] Reserve nodes with accelerator nodes.  ray#43079

[kevin85421]

#2999 (comment)
#2999 (comment)

[ryanaoleary]

Tracking issue for the e2e upgrade tests: #2561

[ryanaoleary]

V2 Autoscaler issues and bug fixes:

• [Autoscaler][V2] Autoscaler V2 does not honor 'Conservative' upscaling mode ray#50259
• [Autoscaler][V2] Updating max replicas while Pods are pending causes v2 autoscaler to hang ray#50868

[rueian]

Add one more issue for autoscaler v2 ray-project/ray#51321.

[bhks]

Hey folks @kevin85421 , @rueian and @ryanaoleary great work here. I know I am late to the party but very interested in joining the team to support. Let me know if any task I can pick up. I have experience in k8s and observability.

Thanks @nadongjun for pointing me to this.

if you guys have any document/RFC/Design I can read that would be great.

[kevin85421]

@bhks Thank you for reaching out! You can check the user guide and the design doc for more details. PRs are welcome. I suggest starting with Autoscaler V2 on KubeRay first. This is a high priority for me at the moment, so related PRs will be reviewed faster. In addition, starting with small PRs makes it easier for them to be merged.

[bhks]

Thank you @kevin85421, Do you have any task in mind which I can start with ?

[rueian]

Add a new one ray-project/ray#52361. I will open a PR soon.