KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Doris-xm · 2025-01-09T14:54:28Z

What this PR does / why we need it:

This PR adds the manifests overlay for Kubeflow Training V2, allowing to install it within Kubeflow Platform.

Which issue(s) this PR fixes :
Fixes #2381

Checklist:

Docs included if any changes are user facing

Electronic-Waste · 2025-01-09T15:25:45Z

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

google-oss-prow · 2025-01-09T15:25:50Z

@Electronic-Waste: GitHub didn't allow me to request PR reviews from the following users: kubeflow/release-team, saileshd1402.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Electronic-Waste

Thanks for creating this @Doris-xm! I left some initial comments for you.

Btw, I recommend that you could learn more about the concept of Training V2. This will help you update the manifests overlay in training-operator and kubeflow/manifests:)

FYI: KubeCon 2024 NA Talk by Andrey and Yuki

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

manifests/v2/overlays/kubeflow/kustomization.yaml

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

Electronic-Waste · 2025-01-10T02:30:03Z

/rerun-all

Electronic-Waste

Basically LGTM. Thank you for your great contributions!

/lgtm
/assign @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads

andreyvelich

Thank you for this great contribution @willb!
/assign @kubeflow/wg-manifests-leads @kubeflow/release-team

manifests/v2/overlays/kubeflow/kustomization.yaml

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

Electronic-Waste

Basically LGTM!

manifests/v2/overlays/kubeflow-platform/kustomization.yaml

Electronic-Waste

Thanks for your great contribution and your patience!

I think, we should also add namespace configuration here:

https://github.com/kubeflow/training-operator/blob/1dfa40c12516fc9eb2ce12c5ef52da7d46670457/manifests/v2/overlays/only-manager/kustomization.yaml#L1-L3

https://github.com/kubeflow/training-operator/blob/1dfa40c12516fc9eb2ce12c5ef52da7d46670457/manifests/v2/overlays/standalone/kustomization.yaml#L1-L3

to install these components in kubeflow-system namespace.

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

…ace: kubeflow-system Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Electronic-Waste

LGTM! Thanks for updating this:)

/lgtm
/assign @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads @saileshd1402

manifests/v2/base/manager/kustomization.yaml

coveralls · 2025-01-20T12:05:01Z

Pull Request Test Coverage Report for Build 12742381626

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 12718184264:	0.0%
Covered Lines:	85
Relevant Lines:	85

💛 - Coveralls

# Conflicts: # manifests/v2/base/manager/kustomization.yaml # manifests/v2/base/rbac/kustomization.yaml # manifests/v2/base/webhook/kustomization.yaml # manifests/v2/overlays/only-manager/kustomization.yaml # manifests/v2/overlays/standalone/kustomization.yaml

Signed-off-by: Xinmin Du <[email protected]>

google-oss-prow · 2025-02-25T14:33:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Electronic-Waste
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Electronic-Waste

Basically LGTM. PTAL if you have time @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads @astefanutti

Btw, can you mark some addressed comments as resolved @Doris-xm ? Some of them are outdated.

/lgtm

juliusvonkohout · 2025-02-26T15:52:08Z

Basically LGTM. PTAL if you have time @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads @astefanutti

Btw, can you mark some addressed comments as resolved @Doris-xm ? Some of them are outdated.

/lgtm

I think when you create the PR against kubeflow/manifests including the integrations tests to see how it behaves with the platform components and authorization, security etc. we can provide more input.

manifests/overlays/kubeflow-platform/kustomization.yaml

manifests/overlays/kubeflow-platform/kubeflow-trainer-roles.yaml

Signed-off-by: Xinmin Du <[email protected]>

Doris-xm · 2025-03-01T16:27:17Z

Thanks for your detailed reviews. Appreciate for your patience and guidance.
I've addressed the comments above. PTAL if you have time:) @kubeflow/wg-training-leads

Electronic-Waste

Sorry for the late review @Doris-xm. Basically LGTM. Thanks!

I think when you create the PR against kubeflow/manifests including the integrations tests to see how it behaves with the platform components and authorization, security etc. we can provide more input.

It will be better if you could adopt the suggestions from @juliusvonkohout and create another PR againset kubeflow/manifests: kubeflow/manifests#2948

/cc @kubeflow/wg-training-leads @astefanutti @kubeflow/wg-manifests-leads

Electronic-Waste · 2025-03-13T12:38:11Z

manifests/overlays/kubeflow-platform/kustomization.yaml

+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kubeflow
+sortOptions:
+  order: fifo
+resources:
+  - ../../base/crds
+  - ../../base/manager
+  - ../../base/rbac
+  - ../../base/runtimes/pretraining


Does FIFO work fine to install all these components? Could you please examine it and provide the screenshot here? @Doris-xm

Thanks for suggestion. I will create a PR in kubeflow/manifests to test it under CI/CD soon.

In my experiment, put runtimes installation before the webhooks works well:

The installation is shown above.
The trainer-controller-manager pod started successfully soon. I took this screenshot before it was ready.
Have I verified all necessary resources? @Electronic-Waste @andreyvelich PTAL if you have time:)

Amazing! That sounds good to me. I have no other comment. Do you have any other ideas? @kubeflow/wg-training-leads @astefanutti

One thing is that, as a required dependency, logically JobSet should be deployed first, so maybe move it in first position of the list, WDYT?

One thing is that, as a required dependency, logically JobSet should be deployed first, so maybe move it in first position of the list, WDYT?

SGTM

google-oss-prow · 2025-03-13T12:44:08Z

@Electronic-Waste: GitHub didn't allow me to request PR reviews from the following users: kubeflow/wg-manifests-leads.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Sorry for the late review @Doris-xm. Basically LGTM. Thanks!

I think when you create the PR against kubeflow/manifests including the integrations tests to see how it behaves with the platform components and authorization, security etc. we can provide more input.

It will be better if you could adopt the suggestions from @juliusvonkohout and create another PR againset kubeflow/manifests: kubeflow/manifests#2948

/cc @kubeflow/wg-training-leads @astefanutti @kubeflow/wg-manifests-leads

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Xinmin Du <[email protected]>

andreyvelich · 2025-03-15T15:57:08Z

manifests/overlays/kubeflow-platform/kustomization.yaml

@@ -4,12 +4,12 @@ namespace: kubeflow
 sortOptions:
  order: fifo


Should we do the same for the standalone installation ?
So we don't need to install manager + runtimes in the separate overlays.

SGTM. Could you please create a separate issue to address it? @Doris-xm

Thanks for suggestion. I start to work on it.

Electronic-Waste · 2025-03-16T05:58:28Z

Thanks for this @Doris-xm! I believe, we can move forward with this PR. In the future, we can raise another PR to integrate trainer into the kubeflow platform: kubeflow/manifests#2948

/lgtm
/assign @kubeflow/wg-training-leads @astefanutti

google-oss-prow bot requested review from jinchihe and kuizhiqing January 9, 2025 14:54

google-oss-prow bot added the size/L label Jan 9, 2025

google-oss-prow bot requested a review from a team January 9, 2025 15:25

google-oss-prow bot added the ok-to-test label Jan 9, 2025

Electronic-Waste reviewed Jan 10, 2025

View reviewed changes

google-oss-prow bot added size/M and removed size/L labels Jan 10, 2025

Electronic-Waste approved these changes Jan 10, 2025

View reviewed changes

google-oss-prow bot assigned Electronic-Waste Jan 10, 2025

google-oss-prow bot added the lgtm label Jan 10, 2025

andreyvelich reviewed Jan 10, 2025

View reviewed changes

manifests/v2/overlays/kubeflow/kustomization.yaml Outdated Show resolved Hide resolved

manifests/v2/overlays/kubeflow/kustomization.yaml Outdated Show resolved Hide resolved

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml Outdated Show resolved Hide resolved

google-oss-prow bot added size/L and removed lgtm size/M labels Jan 12, 2025

Electronic-Waste reviewed Jan 12, 2025

View reviewed changes

manifests/v2/overlays/kubeflow-platform/kustomization.yaml Outdated Show resolved Hide resolved

Electronic-Waste reviewed Jan 13, 2025

View reviewed changes

Xinmin Du added 5 commits January 13, 2025 15:16

Add the manifests overlay for Kubeflow Training V2

a46294f

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: adjust permissions, and format changes

963bfbc

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: rename overlay, adjust event permissions

1100986

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: make namespace configurable

a038ef2

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: move standalone, only-manager installation in namesp…

1f1b0c2

…ace: kubeflow-system Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Doris-xm force-pushed the add-overlay-manifest-v2 branch from 9518f7b to 1f1b0c2 Compare January 13, 2025 07:17

Electronic-Waste approved these changes Jan 13, 2025

View reviewed changes

google-oss-prow bot added the lgtm label Jan 13, 2025

andreyvelich reviewed Jan 13, 2025

View reviewed changes

manifests/v2/base/manager/kustomization.yaml Outdated Show resolved Hide resolved

andreyvelich changed the title ~~Add the manifests overlay for Kubeflow Training V2~~ KEP-2170: Add the manifests overlay for Kubeflow Training V2 Jan 14, 2025

Doris-xm mentioned this pull request Jan 15, 2025

Add Doris-xm as a member kubeflow/internal-acls#733

Merged

Electronic-Waste mentioned this pull request Jan 27, 2025

KEP-2170: Deploy JobSet in kubeflow-system namespace #2388

Merged

Doris-xm added 2 commits February 25, 2025 21:21

Update manifest: add overlay for Kubeflow Platform installation

6509239

Signed-off-by: Xinmin Du <[email protected]>

google-oss-prow bot removed the lgtm label Feb 25, 2025

Electronic-Waste reviewed Feb 26, 2025

View reviewed changes

google-oss-prow bot added the lgtm label Feb 26, 2025

andreyvelich reviewed Feb 26, 2025

View reviewed changes

manifests/overlays/kubeflow-platform/kustomization.yaml Show resolved Hide resolved

manifests/overlays/kubeflow-platform/kubeflow-trainer-roles.yaml Outdated Show resolved Hide resolved

manifests/overlays/kubeflow-platform/kubeflow-trainer-roles.yaml Show resolved Hide resolved

add permission for pods log read & rm persistentvolumeclaims

7a33c55

Signed-off-by: Xinmin Du <[email protected]>

google-oss-prow bot removed the lgtm label Feb 26, 2025

Doris-xm added 2 commits February 27, 2025 00:52

create the runtimes before the webhooks

95788eb

Signed-off-by: Xinmin Du <[email protected]>

Specify sorting order: fifo

32a3da6

Signed-off-by: Xinmin Du <[email protected]>

andreyvelich mentioned this pull request Mar 6, 2025

Add Helm chart for kubeflow trainer #2435

Open

1 task

Electronic-Waste reviewed Mar 13, 2025

View reviewed changes

google-oss-prow bot requested review from a team and astefanutti March 13, 2025 12:44

Deploy jobset first

c61dfe1

Signed-off-by: Xinmin Du <[email protected]>

andreyvelich reviewed Mar 15, 2025

View reviewed changes

This was referenced Mar 16, 2025

KEP-2170: Add manifest overlays for standalone installation #2526

Open

KEP-2170: Add manifest overlays for standalone installation #2527

Open

google-oss-prow bot added the lgtm label Mar 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Doris-xm commented Jan 9, 2025

Electronic-Waste commented Jan 9, 2025

google-oss-prow bot commented Jan 9, 2025

Electronic-Waste left a comment

Electronic-Waste commented Jan 10, 2025

Electronic-Waste left a comment

andreyvelich left a comment

Electronic-Waste left a comment

Electronic-Waste left a comment

Electronic-Waste left a comment

coveralls commented Jan 20, 2025

google-oss-prow bot commented Feb 25, 2025

Electronic-Waste left a comment

juliusvonkohout commented Feb 26, 2025

Doris-xm commented Mar 1, 2025

Electronic-Waste left a comment

Electronic-Waste Mar 13, 2025 •

edited

Loading

Doris-xm Mar 14, 2025 •

edited

Loading

Electronic-Waste Mar 14, 2025

astefanutti Mar 14, 2025

Electronic-Waste Mar 14, 2025

google-oss-prow bot commented Mar 13, 2025

andreyvelich Mar 15, 2025

Electronic-Waste Mar 15, 2025

Doris-xm Mar 16, 2025

Electronic-Waste commented Mar 16, 2025

		@@ -4,12 +4,12 @@ namespace: kubeflow
		sortOptions:
		order: fifo

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Are you sure you want to change the base?

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Conversation

Doris-xm commented Jan 9, 2025

Electronic-Waste commented Jan 9, 2025

google-oss-prow bot commented Jan 9, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

Electronic-Waste commented Jan 10, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

andreyvelich left a comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

coveralls commented Jan 20, 2025

Pull Request Test Coverage Report for Build 12742381626

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

google-oss-prow bot commented Feb 25, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

juliusvonkohout commented Feb 26, 2025

Doris-xm commented Mar 1, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

Electronic-Waste Mar 13, 2025 • edited Loading

Choose a reason for hiding this comment

Doris-xm Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

Electronic-Waste Mar 14, 2025

Choose a reason for hiding this comment

astefanutti Mar 14, 2025

Choose a reason for hiding this comment

Electronic-Waste Mar 14, 2025

Choose a reason for hiding this comment

google-oss-prow bot commented Mar 13, 2025

andreyvelich Mar 15, 2025

Choose a reason for hiding this comment

Electronic-Waste Mar 15, 2025

Choose a reason for hiding this comment

Doris-xm Mar 16, 2025

Choose a reason for hiding this comment

Electronic-Waste commented Mar 16, 2025

Electronic-Waste Mar 13, 2025 •

edited

Loading

Doris-xm Mar 14, 2025 •

edited

Loading