Integrate Azure Machine Learning Operators in Azure Providers

[edgBR]

Description

Currently the Azure Machine Learning provider is available as a completely separated package that is not being maintained actively. This is not the case of sagemaker (where this is part of the main AWS provider) neither Vertex AI (which is part of the main GCP provider).

I am very willing to contribute to this but I need commitment from someone to the community to get this flowing.

Use case/motivation

Azure Machine Learning is a great platform to externalize compute specially using Low Priority instances with GPUs. For teams who do not want to bother managing their own AKS spot pools as well as all the infrastructure required to allocate GPU nodes, orchestrating AML Jobs via Airflow is a great way to lower TCO and focus in what matters.

Besides, Airflow offer providers for similar platforms like AWS Sagemaker, GCP Vertex AI with better support than AzureML. We believe is important to keep this support alive and also to be enhanced.

Related issues

Azure/airflow-provider-azure-machinelearning#16

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

I suggest to contact Microsoft and ask them to contribute the integration.
Airflow has a shared governance model for providers and normally integrations are donated and maintained by the interest party (in that case Microsoft).

[edgBR]

Hi @eladkal

Can you help me here? I have been trying to merge a PR to the provider repo for months but no one answered. I tried to reach the mantainer via linkeding but he also did not answer.

Do you have any contact point I can reach?