Code

AWS ECS Executor#34381

Code

Merged

eladkal merged 62 commits intoapache:mainapache/airflow:mainfrom aws-mwaa:aws_executors/ecsaws-mwaa/upstream-to-airflow:aws_executors/ecsCopy head branch name to clipboardOct 25, 2023

Commits

Commits on Sep 13, 2023

• Introduce AWS Executors to the Amazon provider package. The EcsFargateExecutor and BatchExecutor launches Airflow tasks on AWS ECS/Fargate service and the AWS Batch service, respectively.

  

  aelzeiny

  authored and

  o-nikolas

  committed
  9ff62d8

  Copy full SHA for 9ff62d8

• Reducing scope to just ECS, following up with Batch executor later

  

  ferruzzi

  authored and

  o-nikolas

  committed
  226e6ff

  Copy full SHA for 226e6ff

• Initial static checks

  Show description for aca382a
  

  ferruzzi

  authored and

  o-nikolas

  committed
  aca382a

  Copy full SHA for aca382a

• Convert provided unit tests from UnitTest to pytest

  Show description for 5e02868
  

  ferruzzi

  authored and

  o-nikolas

  committed
  5e02868

  Copy full SHA for 5e02868

• Docstring fixes

  

  ferruzzi

  authored and

  o-nikolas

  committed
  22274ea

  Copy full SHA for 22274ea

• Nested ECS Executor to allow future executors to be added easier

  Show description for ca2b7e1
  

  ferruzzi

  authored and

  o-nikolas

  committed
  ca2b7e1

  Copy full SHA for ca2b7e1

• Remove references to "Fargate" where it wasn't necessary and correct references to ECS which meant EC2

  Show description for 62b14cc
  

  ferruzzi

  authored and

  o-nikolas

  committed
  62b14cc

  Copy full SHA for 62b14cc

• Rewrite TestEcsTaskCollection Tests

  

  ferruzzi

  authored and

  o-nikolas

  committed
  b7457c0

  Copy full SHA for b7457c0

• Remove botocore helpers

  Show description for 575387c
  

  ferruzzi

  authored and

  o-nikolas

  committed
  575387c

  Copy full SHA for 575387c

• Break anything not explicitly Executor related into other modules

  

  ferruzzi

  authored and

  o-nikolas

  committed
  20a0c7c

  Copy full SHA for 20a0c7c

• Add env var to configure logging for container executors

  Show description for a5eef70
  o-nikolas

  committed
  a5eef70

  Copy full SHA for a5eef70

• Changes to the executor config files

  Show description for 455296a
  

  ferruzzi

  authored and

  o-nikolas

  committed
  455296a

  Copy full SHA for 455296a

• Add unit test for config default values

  

  ferruzzi

  authored and

  o-nikolas

  committed
  3bbc42a

  Copy full SHA for 3bbc42a

• Optimize import speed of ECS Executor

  Show description for 99b5d7c
  o-nikolas

  committed
  99b5d7c

  Copy full SHA for 99b5d7c

• First draft for ECS Executor README

  Show description for 466bcc7
  o-nikolas

  committed
  466bcc7

  Copy full SHA for 466bcc7

• Improve unit test code coverage up to 98%

  

  ferruzzi

  authored and

  o-nikolas

  committed
  7847c85

  Copy full SHA for 7847c85

• Convert to using EcsHook

  Show description for dca2837
  

  ferruzzi

  authored and

  o-nikolas

  committed
  dca2837

  Copy full SHA for dca2837

• Add a link to the airflow doc about setting config options

  Show description for 08089ea
  o-nikolas

  committed
  08089ea

  Copy full SHA for 08089ea

• Fix broken subnet config test

  Show description for d0cdc26
  o-nikolas

  committed
  d0cdc26

  Copy full SHA for d0cdc26

• Fix ecs attribute on ECS Executor

  Show description for 3e6a854
  o-nikolas

  committed
  3e6a854

  Copy full SHA for 3e6a854

• Replace mock executor in unit tests

  

  ferruzzi

  authored and

  o-nikolas

  committed
  94be3b6

  Copy full SHA for 94be3b6

• fix typo in docstring

  

  ferruzzi

  authored and

  o-nikolas

  committed
  50e5404

  Copy full SHA for 50e5404

• Add config options to the readme

  

  ferruzzi

  authored and

  o-nikolas

  committed
  dd700bb

  Copy full SHA for dd700bb

• Add a Dockerfile to build the ECS image that will run Airflow tasks

  Show description for 7d02806
  

  syedahsn

  authored and

  o-nikolas

  committed
  7d02806

  Copy full SHA for 7d02806

• Fixes for README Dockerfile static check failures

  o-nikolas

  committed
  23db81f

  Copy full SHA for 23db81f

• Update the order the options are applied: default < template < explicit

  Show description for 6dc7144
  

  ferruzzi

  authored and

  o-nikolas

  committed
  6dc7144

  Copy full SHA for 6dc7144

• Catch all exceptions in methods that interface with the scheduler

  Show description for de69f17
  o-nikolas

  committed
  de69f17

  Copy full SHA for de69f17

• Add guide to set up ECS Executors

  

  syedahsn

  authored and

  o-nikolas

  committed
  4b8365e

  Copy full SHA for 4b8365e

• Fixes to ecs executor config handling

  Show description for 5d6d233
  o-nikolas

  committed
  5d6d233

  Copy full SHA for 5d6d233

• Add instructions on how to check Python version for Airflow image.

  

  syedahsn

  authored and

  o-nikolas

  committed
  5b02b5b

  Copy full SHA for 5b02b5b

• Add intro section to the README

  Show description for 16a1b82
  o-nikolas

  committed
  16a1b82

  Copy full SHA for 16a1b82

• Setup guide fixes

  

  ferruzzi

  authored and

  o-nikolas

  committed
  3a8895f

  Copy full SHA for 3a8895f

• Add performance section to the README

  o-nikolas

  committed
  2ced018

  Copy full SHA for 2ced018

• Update boto user agent to detect Executors as Callers

  o-nikolas

  committed
  c56c096

  Copy full SHA for c56c096

• Allow the executor to fail tasks if they are consistently failing

  

  syedahsn

  authored and

  o-nikolas

  committed
  3766091

  Copy full SHA for 3766091

• fix typo

  

  ferruzzi

  authored and

  o-nikolas

  committed
  3c1d54e

  Copy full SHA for 3c1d54e

• fix whitespace issues

  

  ferruzzi

  authored and

  o-nikolas

  committed
  2d7d1fd

  Copy full SHA for 2d7d1fd

• Allow python dependencies to be installed on the image

  Show description for 4c553af
  

  syedahsn

  authored and

  o-nikolas

  committed
  4c553af

  Copy full SHA for 4c553af

Commits on Sep 14, 2023

• Fix whitespaces in ECS executor test module

  o-nikolas

  committed
  29ff7eb

  Copy full SHA for 29ff7eb

Commits on Sep 15, 2023

• Fixes for build failures

  o-nikolas

  committed
  603367a

  Copy full SHA for 603367a

• Update airflow/providers/amazon/aws/executors/ecs/Setup_guide.md

  Show description for 162b13b
  

  o-nikolas

  and

  jedcunningham

  authored
  162b13b

  Copy full SHA for 162b13b

Commits on Sep 25, 2023

• Update airflow/providers/amazon/aws/executors/ecs/ecs_executor_config.py

  Show description for 2a3119f
  

  o-nikolas

  and

  Taragolis

  authored
  2a3119f

  Copy full SHA for 2a3119f

Commits on Sep 26, 2023

• The max retries config is not a ECS run task kwarg

  o-nikolas

  committed
  3de720f

  Copy full SHA for 3de720f

• Dockerfile bugfix

  Show description for 5fb00f0
  ferruzzi

  committed
  5fb00f0

  Copy full SHA for 5fb00f0

Commits on Oct 4, 2023

• Rename region to region_name

  ferruzzi

  committed
  d2d8db3

  Copy full SHA for d2d8db3

Commits on Oct 5, 2023

• Update the version_added fields in config.yml

  ferruzzi

  committed
  dcca29e

  Copy full SHA for dcca29e

Commits on Oct 10, 2023

• fix string

  ferruzzi

  committed
  6c8cb37

  Copy full SHA for 6c8cb37

Commits on Oct 11, 2023

• Suggested changes from reviewers

  syedahsn

  committed
  0e3b9ea

  Copy full SHA for 0e3b9ea

Commits on Oct 12, 2023

• Merge branch 'main' into aws_executors/ecs

  o-nikolas

  authored
  008695b

  Copy full SHA for 008695b

Commits on Oct 16, 2023

• Use inflection.camelize instead of custom helper

  ferruzzi

  committed
  40f0458

  Copy full SHA for 40f0458

Commits on Oct 19, 2023

• Move config into provider.yaml

  ferruzzi

  committed
  7abbb97

  Copy full SHA for 7abbb97

• Changes for converting docs to rs

  syedahsn

  committed
  d507073

  Copy full SHA for d507073

Commits on Oct 20, 2023

• Add top level Executors index page to amazon docs

  o-nikolas

  committed
  e25b084

  Copy full SHA for e25b084

• Update version_added

  ferruzzi

  committed
  5d42a93

  Copy full SHA for 5d42a93

Commits on Oct 21, 2023

• Merge branch 'main' into aws_executors/ecs

  Show description for 8945379
  ferruzzi

  committed
  8945379

  Copy full SHA for 8945379

Commits on Oct 23, 2023

• typos

  ferruzzi

  committed
  b939730

  Copy full SHA for b939730

• Update doc to clarify consistent configuration

  o-nikolas

  committed
  a8cdf1a

  Copy full SHA for a8cdf1a

Commits on Oct 24, 2023

• Fix config loading

  ferruzzi

  committed
  bcaf9de

  Copy full SHA for bcaf9de

• Merge branch 'main' into aws_executors/ecs

  ferruzzi

  authored
  59e1f6f

  Copy full SHA for 59e1f6f

• Missed a region --> region_name refactor in base aws hook tests

  o-nikolas

  committed
  e5409c5

  Copy full SHA for e5409c5

• Remove Setup_guide readme, the contents have been moved to a docs rst

  o-nikolas

  committed
  bdfd8be

  Copy full SHA for bdfd8be

• No ecs executor configs are sensitive, exclude auto detected kwargs conf

  o-nikolas

  committed
  5e44514

  Copy full SHA for 5e44514