[Epic] [Feature] NetworkPolicy support follow-ups

[machichima]

Related issue

#3987

Descriptions

The main network policy support PR was merged with some follow-ups left #4638

Docs

• Update the docs to recommend a namespace-level default-deny-all policy as a baseline. Since pods and NetworkPolicies are created asynchronously by different controllers, there is a small race window where pods may start before the policy is applied. A pre-existing namespace default-deny policy closes that gap, while the generated allow rules remain additive on top. (link)
• Document a recommendation for users to deploy the kuberay operator in the ray-system namespace instead of the current default
  namespace, to reduce the security surface. (link)
• Provide an example that allows traffic from kube-apiserver or related proxies (link)

Tests

• Add NetworkPolicy e2e coverage on a Kind v1.35 cluster (Kindnet has enforced NetworkPolicy since v1.32): (link)
  • RayJob without clusterSelector, using runtimeEnv
  • RayJob with clusterSelector
  • RayService (verify that we actually receive a response from the Serve application)

Samples

• Fix the CNI-related description in ray-operator/config/samples/ray-cluster.network-isolation-deny-all.yaml. (link)

[chipspeak]

cc @Nolbag @DavidAdaRH

[sanskar-singh-2403]

@machichima can i work on this?

[machichima]

@machichima can i work on this?

@sanskar-singh-2403
Let's check with @chipspeak as he mentioned that his team would like to work on this. Will let you know if there's something that you can take.

[machichima]

@sanskar-singh-2403 Which one would you like to try? Docs part would probably handled by @chipspeak 's team as they contributed this feature. Feel free to comment here!

[chipspeak]

@sanskar-singh-2403 Which one would you like to try? Docs part would probably handled by @chipspeak 's team as they contributed this feature. Feel free to comment here!

@machichima would you prefer to see all 3 docs followups in a single PR under "user guides" or would it be better to split them up? Happy to leave the rest up for grabs @sanskar-singh-2403 !

[sanskar-singh-2403]

@sanskar-singh-2403 Which one would you like to try? Docs part would probably handled by @chipspeak 's team as they contributed this feature. Feel free to comment here!

Thanks @machichima @chipspeak! Since the docs are covered, I'd like to take the Tests and Samples items:

• Add the NetworkPolicy e2e coverage on a Kind v1.35 cluster (RayJob w/o clusterSelector via runtimeEnv, RayJob with clusterSelector, and RayService end-to-end Serve response check).

• Fix the CNI-related description in ray-operator/config/samples/ray-cluster.network-isolation-deny-all.yaml

[machichima]

would you prefer to see all 3 docs followups in a single PR under "user guides" or would it be better to split them up?

Point 1 and 2 should be in the main user guide section, telling user the "best practice" of using network policy in KubeRay, so I think these two can be in the same PR.
For point 3 (Provide an example that allows traffic from kube-apiserver or related proxies ), We can add extra section in the bottom, so a follow-up is ok for this.

[machichima]

Thanks @machichima @chipspeak! Since the docs are covered, I'd like to take the Tests and Samples items

Thanks! Let's go one at a time. I assume you want to start with the e2e test? I'd like to keep the "CNI-related description" item open for other contributors in the meantime. Is that OK?

[sanskar-singh-2403]

ok sure

I'd like to keep the "CNI-related description" item open for other contributors in the meantime. Is that OK?

[chipspeak]

would you prefer to see all 3 docs followups in a single PR under "user guides" or would it be better to split them up?

Point 1 and 2 should be in the main user guide section, telling user the "best practice" of using network policy in KubeRay, so I think these two can be in the same PR. For point 3 (Provide an example that allows traffic from kube-apiserver or related proxies ), We can add extra section in the bottom, so a follow-up is ok for this.

Hey sorry we'd started before I'd seen this so we have all 3 in the one draft PR for now but can adjust as needed!

[sanskar-singh-2403]

Hey @machichima, I've added the e2e tests I could come up with covering the three cases from the issue (RayJob with/without clusterSelector, and RayService). Let me know if you need more.