Skip to content

Commit

Permalink
Update Flyte components (#3927)
Browse files Browse the repository at this point in the history 
* Update Flyte Components

Signed-off-by: Flyte-Bot <[email protected]>

* Add changelog and change to conf.py

Signed-off-by: Eduardo Apolinario <[email protected]>

* Update sandbox-bundled manifest

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Flyte-Bot <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: eapolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
  • Loading branch information
@flyte-bot @eapolinario
3 people authored Aug 11, 2023
1 parent 063a77b commit e6fa812
Show file tree
Hide file tree
Showing 33 changed files with 1,566 additions and 1,461 deletions.
129 changes: 129 additions & 0 deletions CHANGELOG/CHANGELOG-v1.9.0.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Flyte v1.9.0 Release

In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags.


### ArrayNode map tasks

ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/control_flow/map_task.html#map-tasks), the only difference being the submodule used to import the `map_task` function.
More explicitly, let's say you have this code:

```python
from typing import List
from flytekit import map_task, task, workflow

@task
def t(a: int) -> int:
...

@workflow
def wf(xs: List[int]) -> List[int]:
return map_task(t)(a=xs)
```

In order to switch to using array node map tasks you should import map_task from the `flytekit.experimental` module like so:

```python
from typing import List
from flytekit import task, workflow
from flytekit.experimental import map_task

@task
def t(a: int) -> int:
...

@workflow
def wf(xs: List[int]) -> List[int]:
return map_task(t)(a=xs)
```


### Execution tags

Execution tags allow users to can discover their executions and other flyte entities more easily, by creating smarter groupings. The feature is described in this [RFC](https://github.com/flyteorg/flyte/blob/master/rfc/system/0001-flyte-execution-tags.md).

As mentioned before, this feature is shipped in an experimental capacity, the idea being that we're going to incorporate the feedback of the community as we iterate. More work is expected to give prominence to the feature in flyteconsole, in the meanwhile, the feature is supported via [Remote](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/remote_access/index.html).


## Flytekit
* Improve error handling in ShellTask by @pradithya in https://github.com/flyteorg/flytekit/pull/1732
* use default settings for timeline deck width by @cosmicBboy in https://github.com/flyteorg/flytekit/pull/1748
* Raise an exception in case of local execution of raw containers tasks by @eapolinario in https://github.com/flyteorg/flytekit/pull/1745
* Update contributing.rst by @eapolinario in https://github.com/flyteorg/flytekit/pull/1753
* Skip problematic pyyaml versions by @eapolinario in https://github.com/flyteorg/flytekit/pull/1752
* Fail CI tests faster by @eapolinario in https://github.com/flyteorg/flytekit/pull/1756
* Run unit tests on macos-latest by @eapolinario in https://github.com/flyteorg/flytekit/pull/1749
* add rdzv_configs to kfpytorch elastic by @Nan2018 in https://github.com/flyteorg/flytekit/pull/1751
* Fix: Disable rich logging handler when env var `FLYTE_SDK_RICH_TRACEBACKS=0` is set by @fg91 in https://github.com/flyteorg/flytekit/pull/1760
* Throw warning for nested @Task functions by @oliverhu in https://github.com/flyteorg/flytekit/pull/1727
* Remove dependency on responses by @honnix in https://github.com/flyteorg/flytekit/pull/1762
* Fix mlflow test error by @pingsutw in https://github.com/flyteorg/flytekit/pull/1766
* Use phrase 'git revision SHA1' in comments and docs by @DavidMertz in https://github.com/flyteorg/flytekit/pull/1761
* Dockerize docs requirements generation by @eapolinario in https://github.com/flyteorg/flytekit/pull/1764
* Restrict grpcio<1.53.1 by @eapolinario in https://github.com/flyteorg/flytekit/pull/1767
* Array node map task by @eapolinario in https://github.com/flyteorg/flytekit/pull/1640
* Add agent ctrl-c handler to call the delete function. (Reupload) by @Future-Outlier in https://github.com/flyteorg/flytekit/pull/1782
* Add tags to execution by @pingsutw in https://github.com/flyteorg/flytekit/pull/1723

## Flyteadmin
* Alter ID sequence to bigint by @honnix in https://github.com/flyteorg/flyteadmin/pull/578
* Remove content md5 requirement by @wild-endeavor in https://github.com/flyteorg/flyteadmin/pull/587
* Propagate request id on incoming and outgoing requests by @EngHabu in https://github.com/flyteorg/flyteadmin/pull/582
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flyteadmin/pull/589
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flyteadmin/pull/594
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flyteadmin/pull/596
* Bumping flytestdlib by @gvashishtha in https://github.com/flyteorg/flyteadmin/pull/597
* Flyte Execution tags by @pingsutw in https://github.com/flyteorg/flyteadmin/pull/571

## Flyteplugins
* Add support for using task execution ID fields in log URI templates by @jeevb in https://github.com/flyteorg/flyteplugins/pull/372
* Fix generate check in CI by @jeevb in https://github.com/flyteorg/flyteplugins/pull/377
* Remove welcomebot from boilerplate by @eapolinario in https://github.com/flyteorg/flyteplugins/pull/375
* Carry over hash value for all literal types in remote caching by @nicholasjng in https://github.com/flyteorg/flyteplugins/pull/378
* Send task execution metadata to out-core plugin by @honnix in https://github.com/flyteorg/flyteplugins/pull/369
* Support gRPC config for agent-service plugin by @honnix in https://github.com/flyteorg/flyteplugins/pull/368
* Use agent as name where it fits by @honnix in https://github.com/flyteorg/flyteplugins/pull/381
* Fix deletion of elastic task resource requests by @fg91 in https://github.com/flyteorg/flyteplugins/pull/379

## Flytepropeller
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flytepropeller/pull/591
* fixing max parallelism by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/594
* Updated flytestdlib 1.0.20 by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/593
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flytepropeller/pull/597
* Instrument ArrayNode by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/550
* make singular unions castable to their underlying type by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/599
* correct propagation of launchplan start error by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/598
* Bumping flytestdlib and stow versions by @gvashishtha in https://github.com/flyteorg/flytepropeller/pull/602
* Update boilerplate version by @flyte-bot in https://github.com/flyteorg/flytepropeller/pull/601
* Bump flyteplugins to 1.1.15 by @eapolinario in https://github.com/flyteorg/flytepropeller/pull/603
* updated flyteplugins to 1.1.16 by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/604

## Flyteconsole
* feat: Add localStorage for selected Project/Domain by @jsonporter in https://github.com/flyteorg/flyteconsole/pull/774
* Fix project selector failing test by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/780
* fix: node executions list going blank by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/788
* fix: launch form fixes by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/785
* chore: fix 404 due to bad state by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/797
* Fix: Launch Form fixes by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/798
* fix: map tasks should report caching status accurately by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/800
* feat: add support from structured datasets by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/801
* Top level masonry refactor by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/771
* fix: Tasks status out of sync by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/802
* chore: fix test_coverage by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/803
* Fix: breadcrumb feature flag priority order by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/804
* chore: fix yarn.lock by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/805
* fix: releases by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/806
* chore: fix release retry by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/807
* fix: specify node18, semantic release with exec and git plugins by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/808
* fix: commit linter running on generated messages by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/810
* chore: remove release git step by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/811
* fix: union value handling in launch form by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/812

## New Contributors
* @Nan2018 made their first contribution in https://github.com/flyteorg/flytekit/pull/1751
* @oliverhu made their first contribution in https://github.com/flyteorg/flytekit/pull/1727
* @DavidMertz made their first contribution in https://github.com/flyteorg/flytekit/pull/1761
* @Future-Outlier made their first contribution in https://github.com/flyteorg/flytekit/pull/1782
* @gvashishtha made their first contribution in https://github.com/flyteorg/flyteadmin/pull/597
* @nicholasjng made their first contribution in https://github.com/flyteorg/flyteplugins/pull/378
* @gvashishtha made their first contribution in https://github.com/flyteorg/flytepropeller/pull/602
2 changes: 1 addition & 1 deletion charts/flyte-binary/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Chart for basic single Flyte executable deployment
| configuration.auth.oidc.clientId | string | `""` | |
| configuration.auth.oidc.clientSecret | string | `""` | |
| configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` | |
| configuration.co-pilot.image.tag | string | `"v0.0.30"` | |
| configuration.co-pilot.image.tag | string | `"v0.0.33"` | |
| configuration.database.dbname | string | `"flyte"` | |
| configuration.database.host | string | `"127.0.0.1"` | |
| configuration.database.options | string | `"sslmode=disable"` | |
Expand Down
2 changes: 1 addition & 1 deletion charts/flyte-binary/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ configuration:
# repository CoPilot sidecar image repository
repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE
# tag CoPilot sidecar image tag
tag: v0.0.30 # FLYTECOPILOT_TAG
tag: v0.0.33 # FLYTECOPILOT_TAG
# agentService Flyte Agent configuration
agentService: {}
# Uncomment and modify to include configuration for Flyte Agent
Expand Down
14 changes: 7 additions & 7 deletions charts/flyte-core/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ helm install gateway bitnami/contour -n flyte
| configmap.clusters.clusterConfigs | list | `[]` | |
| configmap.clusters.labelClusterMap | object | `{}` | |
| configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.32","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.32","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.33","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.33","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration |
| configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). |
| configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). |
Expand Down Expand Up @@ -119,7 +119,7 @@ helm install gateway bitnami/contour -n flyte
| datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment |
| datacatalog.image.tag | string | `"v1.0.48"` | Docker image tag |
| datacatalog.image.tag | string | `"v1.0.51"` | Docker image tag |
| datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment |
| datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods |
| datacatalog.priorityClassName | string | `""` | Sets priorityClassName for datacatalog pod(s). |
Expand Down Expand Up @@ -151,7 +151,7 @@ helm install gateway bitnami/contour -n flyte
| flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command |
| flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | |
| flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment |
| flyteadmin.image.tag | string | `"v1.1.110"` | |
| flyteadmin.image.tag | string | `"v1.1.118"` | |
| flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create |
| flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment |
| flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods |
Expand Down Expand Up @@ -196,7 +196,7 @@ helm install gateway bitnami/contour -n flyte
| flyteconsole.ga.tracking_id | string | `"G-0QW4DJWJ20"` | |
| flyteconsole.image.pullPolicy | string | `"IfNotPresent"` | |
| flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment |
| flyteconsole.image.tag | string | `"v1.8.5"` | |
| flyteconsole.image.tag | string | `"v1.9.2"` | |
| flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment |
| flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods |
| flyteconsole.priorityClassName | string | `""` | Sets priorityClassName for flyte console pod(s). |
Expand All @@ -216,7 +216,7 @@ helm install gateway bitnami/contour -n flyte
| flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | |
| flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment |
| flytepropeller.image.tag | string | `"v1.1.106"` | |
| flytepropeller.image.tag | string | `"v1.1.116"` | |
| flytepropeller.manager | bool | `false` | |
| flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment |
| flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods |
Expand All @@ -236,7 +236,7 @@ helm install gateway bitnami/contour -n flyte
| flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files |
| flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment |
| flytescheduler.image.tag | string | `"v1.1.110"` | Docker image tag |
| flytescheduler.image.tag | string | `"v1.1.118"` | Docker image tag |
| flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment |
| flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods |
| flytescheduler.priorityClassName | string | `""` | Sets priorityClassName for flyte scheduler pod(s). |
Expand Down
12 changes: 6 additions & 6 deletions charts/flyte-core/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ flyteadmin:
image:
# -- Docker image for Flyteadmin deployment
repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE
tag: v1.1.110 # FLYTEADMIN_TAG
tag: v1.1.118 # FLYTEADMIN_TAG
pullPolicy: IfNotPresent
# -- Additional flyteadmin container environment variables
#
Expand Down Expand Up @@ -97,7 +97,7 @@ flytescheduler:
# -- Docker image for Flytescheduler deployment
repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE
# -- Docker image tag
tag: v1.1.110 # FLYTESCHEDULER_TAG
tag: v1.1.118 # FLYTESCHEDULER_TAG
# -- Docker image pull policy
pullPolicy: IfNotPresent
# -- Default resources requests and limits for Flytescheduler deployment
Expand Down Expand Up @@ -151,7 +151,7 @@ datacatalog:
# -- Docker image for Datacatalog deployment
repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE
# -- Docker image tag
tag: v1.0.48 # DATACATALOG_TAG
tag: v1.0.51 # DATACATALOG_TAG
# -- Docker image pull policy
pullPolicy: IfNotPresent
# -- Default resources requests and limits for Datacatalog deployment
Expand Down Expand Up @@ -274,7 +274,7 @@ flytepropeller:
image:
# -- Docker image for Flytepropeller deployment
repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE
tag: v1.1.106 # FLYTEPROPELLER_TAG
tag: v1.1.116 # FLYTEPROPELLER_TAG
pullPolicy: IfNotPresent
# -- Default resources requests and limits for Flytepropeller deployment
resources:
Expand Down Expand Up @@ -332,7 +332,7 @@ flyteconsole:
image:
# -- Docker image for Flyteconsole deployment
repository: cr.flyte.org/flyteorg/flyteconsole # FLYTECONSOLE_IMAGE
tag: v1.8.5 # FLYTECONSOLE_TAG
tag: v1.9.2 # FLYTECONSOLE_TAG
pullPolicy: IfNotPresent
# -- Default resources requests and limits for Flyteconsole deployment
resources:
Expand Down Expand Up @@ -657,7 +657,7 @@ configmap:
# -- Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig)
co-pilot:
name: flyte-copilot-
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.32 # FLYTECOPILOT_IMAGE
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.33 # FLYTECOPILOT_IMAGE
start-timeout: 30s

# -- Core propeller configuration
Expand Down
Loading

0 comments on commit e6fa812

Please sign in to comment.