Flyte multi-cluster deployment - Allow deferring cluster label selection to external service

[fg91]

Flyte can be deployed with multiple data plane clusters in which executions can be scheduled.

Which cluster is used for an execution can be selected in various ways:

• Default execution cluster label configured in flyteadmin config (see here)
• Execution cluster label set via matchable attributes on a project, domain or workflow level (see here)
• Pass the execution cluster label as a cli arg to pyflyte run

---

As a platform engineer maintaining Flyte, I would like to 1) relieve platform users from having to decide on the execution cluster and 2) take realtime consumption of resources into account in an automated way when deciding which cluster to use.

• I propose to define a gRPC service that flyteadmin can be configured to call to decide on an execution cluster label.
• I propose that it would be the users' responsibility to implement this service for their respective cloud provider/infrastructure and their specific needs.

The service that the user would implement for themselves could e.g. monitor utilization of the available clusters and current cloud provider resource quotas in order to decide on an execution cluster label.

---

As a bonus feature, flyteadmin could provide this external service with a summary which resource types (in particular accelerator types) will be used in what quantity by the workflow execution. This would be very helpful but not a requirement.

[fg91]

Update contributors' sync:
Turn into short RFC to agree on interface.