[CI] Deflaky TestRayServiceSuspendDuringIncrementalUpgrade

[win5923]

Description

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/15947/canvas?jid=019f8226-29f2-48d0-98f2-e5c6c1f4c851&tab=artifacts

The e2e test TestRayServiceSuspendDuringIncrementalUpgrade is flaky. It intermittently fails when the test flips Spec.Suspend on the RayService, with a 409 Conflict:

    rayservice_suspend_test.go:67: 
        Unexpected error:
            <*errors.StatusError | 0xe8d925d2b40>: 
            Operation cannot be fulfilled on rayservices.ray.io "suspend-incremental-rayservice": the object has been modified; please apply your changes to the latest version and try again
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                        ShardInfo: nil,
                    },
                    Status: "Failure",
                    Message: "Operation cannot be fulfilled on rayservices.ray.io \"suspend-incremental-rayservice\": the object has been modified; please apply your changes to the latest version and try again",
                    Reason: "Conflict",
                    Details: {
                        Name: "suspend-incremental-rayservice",
                        Group: "ray.io",
                        Kind: "rayservices",
                        UID: "",
                        Causes: nil,
                        RetryAfterSeconds: 0,
                    },
                    Code: 409,
                },
            }
        occurred

kuberay/ray-operator/test/e2erayservice/rayservice_suspend_test.go

Lines 44 to 49 in 7af74cb

	LogWithTimestamp(test.T(), "Setting Spec.Suspend=true")
	rayService, err = GetRayService(test, namespace.Name, rayServiceName)
	g.Expect(err).NotTo(HaveOccurred())
	rayService.Spec.Suspend = true
	_, err = test.Client().Ray().RayV1().RayServices(namespace.Name).Update(test.Ctx(), rayService, metav1.UpdateOptions{})
	g.Expect(err).NotTo(HaveOccurred())

According to #4915 (review), we should wrap both read-modify-write blocks in Eventually to prevent 409 error.

[win5923]

Also in rayservice_incremental_upgrade_test.go has similar issues.

kuberay/ray-operator/test/e2eincrementalupgrade/rayservice_incremental_upgrade_test.go

Lines 369 to 375 in 7af74cb

	// Trigger an incremental upgrade through a change to the RayCluster spec.
	LogWithTimestamp(test.T(), "Triggering an upgrade for RayService %s/%s", rayService.Namespace, rayService.Name)
	rayService, err = GetRayService(test, namespace.Name, rayServiceName)
	g.Expect(err).NotTo(HaveOccurred())
	rayService.Spec.RayClusterSpec.WorkerGroupSpecs[0].Template.Spec.Containers[0].Resources.Requests[corev1.ResourceCPU] = resource.MustParse("500m")
	_, err = test.Client().Ray().RayV1().RayServices(namespace.Name).Update(test.Ctx(), rayService, metav1.UpdateOptions{})
	g.Expect(err).NotTo(HaveOccurred())

[yjaw]

@win5923 I can help with this!

[400Ping]

@yjaw nice, thanks for taking up the issue!