Flyte v1.10.6 milestone release

[flyte-bot]

Flyte 1.10.6 Release

Due to a mishap in the move to the monorepo, we ended up generating the git tags between 1.10.1 to 1.10.5, so in order to decrease the confusion we decided to skip those patch versions and go straight to the next available version.

We've shipped a ton of stuff in this patch release, here are some of the highlights.

GPU Accelerators

You'll be able to get more fine-grained in the use GPU Accelerators in your tasks. Here are some examples:

No preference of GPU accelerator to use:

@task(limits=Resources(gpu="1"))
def my_task() -> None:
    ...

Schedule on a specific GPU accelerator:

from flytekit.extras.accelerators import T4


@task(
    limits=Resources(gpu="1"),
    accelerator=T4,
)
def my_task() -> None:
    ...

Schedule on a Multi-instance GPU (MIG) accelerator with no preference of partition size:

from flytekit.extras.accelerators import A100


@task(
    limits=Resources(gpu="1"),
    accelerator=A100,
)
def my_task() -> None:
    ...

Schedule on a Multi-instance GPU (MIG) accelerator with a specific partition size:

from flytekit.extras.accelerators import A100


@task(
    limits=Resources(gpu="1"),
    accelerator=A100.partition_1g_5gb,
)
def my_task() -> None:
    ...

Schedule on an unpartitioned Multi-instance GPU (MIG) accelerator:

from flytekit.extras.accelerators import A100


@task(
    limits=Resources(gpu="1"),
    accelerator=A100.unpartitioned,
)
def my_task() -> None:
    ...

Improved support for Ray logs

#4266 opens the door for RayJob logs to be persisted.

In #4397 we added support for a link to a Ray dashboard to show up in the task card.

Updated grafana dashboards

We updated the official grafana dashboards in #4382.

Support for Azure AD

A new version of our stow fork added support for Azure AD in flyteorg/stow#9.

Full changelog:

• Restructure Flyte releases by @eapolinario in Restructure Flyte releases #4304
• Use debian bookworm as single binary base image by @eapolinario in Use debian bookworm as single binary base image  #4311
• Use local version in single-binary by @eapolinario in Use local version in single-binary #4294
• Accessibility for README by @mishmanners in Accessibility for README #4322
• Add tests in flytepropeller/pkg /controller/executors from 72.3% to 87.3% coverage by @Future-Outlier in Add tests in flytepropeller/pkg /controller/executors from 72.3% to 87.3% coverage #4276
• fix: remove unused setting in deployment charts by @HeetVekariya in fix: remove unused setting in deployment charts #4252
• Document simplified retry behaviour introduced in RFC: Simplify retry behaviour and unify it between plugins #3902 by @fg91 in Document simplified retry behaviour introduced in #3902 #4022
• Ray logs persistence by @jeevb in Ray logs persistence #4266
• Not revisiting task nodes and correctly incrementing parallelism by @hamersaw in Not revisiting task nodes and correctly incrementing parallelism #4318
• Fix RunPluginEndToEndTest util by @andresgomezfrr in Fix RunPluginEndToEndTest util #4342
• Tune sandbox readiness checks to ensure that sandbox is fully accessi… by @jeevb in Tune sandbox readiness checks to ensure that sandbox is fully accessi… #4348
• Chore: Ensure Stalebot doesn't close issues we've not yet triaged. by @brndnblck in Chore: Ensure Stalebot doesn't close issues we've not yet triaged. #4352
• Do not automatically close stale issues by @eapolinario in Do not automatically close stale issues #4353
• Fix: Set flyteadmin gRPC port to 80 in ingress if using TLS between load balancer and backend by @fg91 in Fix: Set flyteadmin gRPC port to 80 in ingress if using TLS between load balancer and backend #3964
• Support Databricks WebAPI 2.1 version and Support existing_cluster_id and new_cluster options to create a Job by @Future-Outlier in Support Databricks WebAPI 2.1 version and Support existing_cluster_id and new_cluster options to create a Job #4361
• Fixing caching on maptasks when using partials by @hamersaw in Fixing caching on maptasks when using partials #4344
• Fix read raw limit by @honnix in Fix read raw limit #4370
• minor fix to eks-starter.yaml by @guyarad in minor fix to eks-starter.yaml #4337
• Reporting running if the primary container status is not yet reported by @hamersaw in Reporting running if the primary container status is not yet reported #4339
• completing retries even if minSuccesses are achieved by @hamersaw in completing retries even if minSuccesses are achieved #4338
• Add comment to auth scope by @wild-endeavor in Add comment to auth scope #4341
• Update tests by @eapolinario in Update tests #4381
• Update order of cluster resources config to work with both uctl and flytectl by @neverett in Update order of cluster resources config to work with both uctl and flytectl #4373
• Update tests in single-binary by @eapolinario in Update tests in single-binary #4383
• Passthrough unique node ID in task execution ID for generating log te… by @jeevb in Passthrough unique node ID in task execution ID for generating log te… #4380
• Add Sections in the PR Template by @Future-Outlier in Add Sections in the PR Template #4367
• Update metadata in ArrayNode TaskExecutionEvents by @hamersaw in Update metadata in ArrayNode TaskExecutionEvents #4355
• Fixes list formatting in flytepropeller arch docs by @thomasjpfan in Fixes list formatting in flytepropeller arch docs #4345
• Update boilerplate end2end tests by @hamersaw in Update boilerplate end2end tests #4393
• Handle all ray job statuses by @EngHabu in Handle all ray job statuses #4389
• Relocate sandbox config by @davidmirror-ops in Relocate sandbox config #4385
• Refactor task logs framework by @jeevb in Refactor task logs framework #4396
• Add support for displaying the Ray dashboard when a RayJob is active by @jeevb in Add support for displaying the Ray dashboard when a RayJob is active #4397
• Disable path filtering for monorepo components by @eapolinario in Disable path filtering for monorepo components #4404
• Silence NotFound when get task resource by @honnix in Silence NotFound when get task resource #4388
• adding consoleUrl parameterization based on partition by @lauralindy in adding consoleUrl parameterization based on partition #4375
• [Docs] Sensor Agent Doc by @Future-Outlier in [Docs] Sensor Agent Doc #4195
• [flytepropeller] Add Tests in v1alpha.go including array_test.go, branch_test.go, error_test.go, and iface_test.go with 0.13% Coverage Improvement by @Future-Outlier in [flytepropeller] Add Tests in v1alpha.go including array_test.go, branch_test.go, error_test.go, and iface_test.go with 0.13% Coverage Improvment #4234
• Add more context for ray log template links by @jeevb in Add more context for ray log template links #4416
• Add ClusterRole config for Ray by @davidmirror-ops in Add ClusterRole config for Ray #4405
• Fix and update scripts for generating grafana dashboards by @Tom-Newton in Fix and update scripts for generating grafana dashboards #4382
• Add artifacts branch to publish to buf on push by @squiishyy in Add artifacts branch to publish to buf on push #4450
• Add service monitor for flyte admin and propeller service by @vraiyaninv in Add service monitor for flyte admin and propeller service #4427
• Fix Kubeflow TF Operator GetTaskPhase Bug by @Future-Outlier in Fix Kubeflow TF Operator GetTaskPhase Bug #4469
• Instrument opentelemetry by @hamersaw in Instrument opentelemetry #4357
• Delete the .github folder from each subdirectory by @pingsutw in Delete the .github folder from each subdirectory #4480
• Fix the loop variable scheduler issue by @pmahindrakar-oss in Fix the loop variable scheduler issue #4468
• Databricks Plugin Setup Doc Enhancement by @Future-Outlier in Databricks Plugin Setup Doc Enhancement #4445
• Put ticker back in place in propeller gc by @eapolinario in Put ticker back in place in propeller gc #4490
• Store failed execution in flyteadmin by @iaroslav-ciupin in Store failed execution in flyteadmin #4390
• Moving from flyteadmin - Upgrade coreos/go-oidc to v3 to pickup claims parsing fixes by @eapolinario in Moving from flyteadmin - Upgrade coreos/go-oidc to v3 to pickup claims parsing fixes #4139
• Bump flyteorg/stow to 0.3.8 by @eapolinario in Bump flyteorg/stow to 0.3.8 #4312
• Remove 'needs' from generate_flyte_manifest by @eapolinario in Remove 'needs' from generate_flyte_manifest #4495
• Update Flyte components by @flyte-bot in Update Flyte components #4302
• Modify how flytecopilot version is parsed from values file by @eapolinario in Modify how flytecopilot version is parsed from values file #4496
• Ignore component tags in goreleaser by @eapolinario in Ignore component tags in goreleaser #4497
• Fix indentation of shell: task by @eapolinario in Fix indentation of shell: task #4498
• Implemented simple echo plugin for testing by @hamersaw in Implemented simple echo plugin for testing #4489
• Correctly handle resource overrides in KF plugins by @jeevb in Correctly handle resource overrides in KF plugins #4467
• Remove deprecated InjectDecoder by @EngHabu in Remove deprecated InjectDecoder #4507
• Fix $HOME resolution and webhook namespace by @EngHabu in Fix $HOME resolution and webhook namespace #4509
• Add note on updating sandbox cluster configuration by @jeevb in Add note on updating sandbox cluster configuration #4510
• Add New PR Template by @Future-Outlier in Add New PR Template #4512
• [Docs] Databricks Agent Doc by @Future-Outlier in [Docs] Databricks Agent Doc #4008
• Bump version of goreleaser gh action to v5 by @eapolinario in Bump version of goreleaser gh action to v5 #4519
• Kf operators use GetReplicaFunc (Error Handling) by @Future-Outlier in Kf operators use GetReplicaFunc (Error Handling) #4471

New Contributors

• @HeetVekariya made their first contribution in fix: remove unused setting in deployment charts #4252
• @andresgomezfrr made their first contribution in Fix RunPluginEndToEndTest util #4342
• @brndnblck made their first contribution in Chore: Ensure Stalebot doesn't close issues we've not yet triaged. #4352
• @guyarad made their first contribution in minor fix to eks-starter.yaml #4337
• @neverett made their first contribution in Update order of cluster resources config to work with both uctl and flytectl #4373
• @thomasjpfan made their first contribution in Fixes list formatting in flytepropeller arch docs #4345
• @lauralindy made their first contribution in adding consoleUrl parameterization based on partition #4375
• @Tom-Newton made their first contribution in Fix and update scripts for generating grafana dashboards #4382
• @vraiyaninv made their first contribution in Add service monitor for flyte admin and propeller service #4427

---

This discussion was created from the release Flyte v1.10.6 milestone release.