[Epic] [Feature] mTLS follow-ups

[machichima]

Related issue

#3974

Description

Based on summary of the previous PR: #4566 (comment)

Docs

Should include following contents

• Document to teach user now to use this feature
• Ray currently does not support certificate renewal (application only read cert when starting up), to renew it, user should restart Ray application
• max num of IPs in cert (default)
  • cert-manager ≥ v1.19 (not configurable before 1.20)
    • maxLeafCertificatePEMSize: 36,500
      • v1.20 can configure through pemSizeLimitsConfig.maxCertificateSize
    • cert-manager leave 30000 for SANs (in real-world it can be bigger than this, this value can be view as the min size we have)
      • https://github.com/cert-manager/cert-manager/blob/ae6723401bd1bef1c00bd3c46a52c15387cd05ba/internal/pem/decode.go#L63-L68
  • Size pr IPv4: 6 bytes DER → PEM base64 ×4/3 ≈ 8.2 bytes/IP
  • 30000 / 8.2 ~ 3,658 IPs (conservative lower bound)

Refactor

1. Extract the part building issuer to a helper function: https://github.com/ray-project/kuberay/pull/4566/changes#r3537218847
   • @vincent-626
2. refactor test https://github.com/ray-project/kuberay/pull/4566/changes#r3544045185

Bug

1. Fix update secret on every reconcile: [Feature] Add mTLS Support via Cert Manager to RayCluster #4566 (comment)
   • @400Ping

e2e test

1. cross node communication: https://github.com/ray-project/kuberay/pull/4566/changes#r3544071464
2. autoscaler + mTLS: current test only checks the sidecar has TLS env vars, we should add a case that triggers a scale-up and verifies new worker pods join.

Future Plan

1. BYOC (bring your own certificate) support -> need further discussion. We can support this if there's user request
2. supports hostname-based identity instead of pod IP until ray-project/ray#61651 is resolved (ref)

[machichima]

cc @chipspeak

[400Ping]

I will like to take the bug one.

[vincent-626]

Hi @machichima I can help with the refactor #1.

[machichima]

Thank you @400Ping @vincent-626 ! Please go ahead

[HsiaoHungKai]

@machichima Can I take refactor #2