Adds comet-ml plugin #2550

thomasjpfan · 2024-07-03T03:02:37Z

Why are the changes needed?

This PR adds a comet-ml plugin to flytekit.

What changes were proposed in this pull request?

This PR adds a comet_ml_init task decorator that adds a link to the Comet's platform.

How was this patch tested?

Unit tests were added to this PR.

Screenshots

Related PRs

Related to #2405

Docs link

I also added an example in flytesnacks: flyteorg/flytesnacks#1702

Signed-off-by: Thomas J. Fan <[email protected]>

codecov · 2024-07-03T03:10:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.41%. Comparing base (876877a) to head (58b33b0).
Report is 20 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2550       +/-   ##
===========================================
+ Coverage   52.89%   90.41%   +37.52%     
===========================================
  Files         297       60      -237     
  Lines       23957     3058    -20899     
  Branches     3655        0     -3655     
===========================================
- Hits        12671     2765     -9906     
+ Misses      11178      293    -10885     
+ Partials      108        0      -108

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

samhita-alla · 2024-07-17T16:35:41Z

plugins/flytekit-comet-ml/flytekitplugins/comet_ml/tracking.py

+            # For local execution, always use the experiment_key. If `self.experiment_key` is `None`, comet_ml
+            # will generate it's own key
+            if self.experiment_key is not None:
+                init_kwargs["experiment_key"] = self.experiment_key


is api key not required for local executions?

It's required, I use export COMET_API_KEY="xxxx" to test it in local execution.

samhita-alla · 2024-07-17T16:39:42Z

plugins/flytekit-comet-ml/flytekitplugins/comet_ml/tracking.py

+            raise ValueError("project_name must be set")
+        if workspace is None:
+            raise ValueError("workspace must be set")
+        if secret is None:


shouldn't secret be made optional as it isn't necessary when executing locally?

The secret field is ignored during location execution, but required during remote. One still needs export COMET_API_KEY= for local execution.

If this was optional, we'll catch the "no secret" during execution and not at registration time. I do not want to wait until then, because these experiments can use GPUs and waiting & spinning up a GPU just to see this error is not efficient and a worst DevX.

By making secret required, I am optimizing for "failing quickly" for remote execution.

Signed-off-by: Thomas J. Fan <[email protected]>

…ml_init Signed-off-by: Thomas J. Fan <[email protected]>

Future-Outlier · 2024-07-17T18:20:41Z

plugins/flytekit-comet-ml/README.md

+    dynamic-log-links:
+      - comet-ml-execution-id:
+          displayName: Comet
+          templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .executionName }}{{ .nodeId }}{{ .taskRetryAttempt }}{{ .taskConfig.link_suffix }}


we should use double quote here, right?

Future-Outlier · 2024-07-17T18:20:51Z

plugins/flytekit-comet-ml/README.md

+          templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .executionName }}{{ .nodeId }}{{ .taskRetryAttempt }}{{ .taskConfig.link_suffix }}
+      - comet-ml-custom-id:
+          displayName: Comet
+          templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .taskConfig.experiment_key }}


Future-Outlier · 2024-07-17T18:28:33Z

plugins/flytekit-comet-ml/flytekitplugins/comet_ml/tracking.py

+        project_name: Optional[str] = None,
+        workspace: Optional[str] = None,
+        experiment_key: Optional[str] = None,
+        secret: Optional[Union[Secret, Callable]] = None,


Can project_name, workspace, and secret be None Optional here in the type hints?
It will be better for the type hints

They are already Optional here. Are you suggesting to not make them Optional?

I got it to work with 417d03b It forces project_name and friends to be required.

Future-Outlier · 2024-07-17T18:53:27Z

@thomasjpfan
Can we add flytekitplugins-comet-ml in the file below?
It will help us run the CI to build the plugin.

https://github.com/flyteorg/flytekit/blob/master/.github/workflows/pythonbuild.yml#L319

Signed-off-by: Thomas J. Fan <[email protected]>

* Adds comet-ml plugin Signed-off-by: Thomas J. Fan <[email protected]> * For local execution, do not set experiment_key if it is none Signed-off-by: Thomas J. Fan <[email protected]> * Use correct comet-ml links Signed-off-by: Thomas J. Fan <[email protected]> * Allow host to be adjustable Signed-off-by: Thomas J. Fan <[email protected]> * Adds comet-ml plugin Signed-off-by: Thomas J. Fan <[email protected]> * Use new comet-ml login name Signed-off-by: Thomas J. Fan <[email protected]> * Require the project_name workspace and secrets Signed-off-by: Thomas J. Fan <[email protected]> --------- Signed-off-by: Thomas J. Fan <[email protected]> Signed-off-by: mao3267 <[email protected]>

thomasjpfan added 5 commits June 26, 2024 13:54

Adds comet-ml plugin

ad70d64

Signed-off-by: Thomas J. Fan <[email protected]>

For local execution, do not set experiment_key if it is none

da94ee2

Signed-off-by: Thomas J. Fan <[email protected]>

Use correct comet-ml links

117f050

Signed-off-by: Thomas J. Fan <[email protected]>

Allow host to be adjustable

4cfa4eb

Signed-off-by: Thomas J. Fan <[email protected]>

Adds comet-ml plugin

58b33b0

Signed-off-by: Thomas J. Fan <[email protected]>

thomasjpfan requested review from wild-endeavor, kumare3, eapolinario, pingsutw, cosmicBboy, samhita-alla and Future-Outlier as code owners July 3, 2024 03:02

thomasjpfan mentioned this pull request Jul 3, 2024

Adds comet-ml plugin example flyteorg/flytesnacks#1702

Open

samhita-alla reviewed Jul 17, 2024

View reviewed changes

thomasjpfan added 2 commits July 17, 2024 12:44

Use new comet-ml login name

d04d012

Signed-off-by: Thomas J. Fan <[email protected]>

Merge remote-tracking branch 'upstream/master' into thomasjpfan/comet…

ead937d

…ml_init Signed-off-by: Thomas J. Fan <[email protected]>

Future-Outlier reviewed Jul 17, 2024

View reviewed changes

Require the project_name workspace and secrets

cc2e9ad

Signed-off-by: Thomas J. Fan <[email protected]>

thomasjpfan force-pushed the thomasjpfan/cometml_init branch from 0eb34fa to cc2e9ad Compare July 17, 2024 20:04

Future-Outlier approved these changes Jul 17, 2024

View reviewed changes

Future-Outlier merged commit 5a4ba2f into flyteorg:master Jul 17, 2024
92 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds comet-ml plugin #2550

Adds comet-ml plugin #2550

thomasjpfan commented Jul 3, 2024 •

edited

Loading

codecov bot commented Jul 3, 2024 •

edited

Loading

samhita-alla Jul 17, 2024

Future-Outlier Jul 17, 2024

samhita-alla Jul 17, 2024

thomasjpfan Jul 17, 2024 •

edited

Loading

Future-Outlier Jul 17, 2024 •

edited

Loading

Future-Outlier Jul 17, 2024

Future-Outlier Jul 17, 2024

thomasjpfan Jul 17, 2024

thomasjpfan Jul 17, 2024

Future-Outlier commented Jul 17, 2024

Adds comet-ml plugin #2550

Adds comet-ml plugin #2550

Conversation

thomasjpfan commented Jul 3, 2024 • edited Loading

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Screenshots

Related PRs

Docs link

codecov bot commented Jul 3, 2024 • edited Loading

Codecov Report

samhita-alla Jul 17, 2024

Choose a reason for hiding this comment

Future-Outlier Jul 17, 2024

Choose a reason for hiding this comment

samhita-alla Jul 17, 2024

Choose a reason for hiding this comment

thomasjpfan Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

Future-Outlier Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

Future-Outlier Jul 17, 2024

Choose a reason for hiding this comment

Future-Outlier Jul 17, 2024

Choose a reason for hiding this comment

thomasjpfan Jul 17, 2024

Choose a reason for hiding this comment

thomasjpfan Jul 17, 2024

Choose a reason for hiding this comment

Future-Outlier commented Jul 17, 2024

thomasjpfan commented Jul 3, 2024 •

edited

Loading

codecov bot commented Jul 3, 2024 •

edited

Loading

thomasjpfan Jul 17, 2024 •

edited

Loading

Future-Outlier Jul 17, 2024 •

edited

Loading