Skip to content

Commit

Permalink
AWS System Test: EMR Container
Browse files Browse the repository at this point in the history
Ensure that the EMR cluster stays running for long enough for all the
tasks in the DAG to complete their operations against the cluster.
  • Loading branch information
o-nikolas committed Sep 19, 2024
1 parent 46b41e3 commit cd5269b
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion tests/system/providers/amazon/aws/example_emr.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,10 @@
"InstanceCount": 1,
},
],
"KeepJobFlowAliveWhenNoSteps": False,
# If the EMR steps complete too quickly the cluster will be torn down before the other system test
# tasks have a chance to run (such as the modify cluster step, the addition of more EMR steps, etc).
# Set KeepJobFlowAliveWhenNoSteps to False to avoid the cluster from being torn down prematurely.
"KeepJobFlowAliveWhenNoSteps": True,
"TerminationProtected": False,
},
"Steps": SPARK_STEPS,
Expand Down

0 comments on commit cd5269b

Please sign in to comment.