Skip to content

Conversation

@LucaCinquini
Copy link
Collaborator

@LucaCinquini LucaCinquini commented Mar 18, 2025

Purpose

  • This PR increases the EC2 resources (memory and CPU) available to the Airflow core components (web server, triggerer, dag-processor, scheduler).
  • It also adds new EC2 instance types to the list of available machines that a user can select
  • Finally, it changes the configuration of the Karpenter node pools to be compatible with all the selectable EC2 types

Proposed Changes

  • [ADD] New EC2 types to the selectable list
  • [CHANGE] Allocation of the Airflow core components pods onto the "airflow-core-components" Karpenter node pool
  • [FIX] Lower and upper limits for some Karpenter node pool configuration
  • [CHANGE] Increase the default values for total number of CPUs and memory for all jobs running concurrently

Issues

Testing

  • Successfully deployed SPS with new configuration.
  • Successfully executed EMIT workflows on EC2s of type "m5ad" and the older types.

@LucaCinquini LucaCinquini marked this pull request as draft March 18, 2025 14:23
@LucaCinquini LucaCinquini marked this pull request as ready for review March 18, 2025 21:45
Copy link
Collaborator

@nikki-t nikki-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me!

@LucaCinquini LucaCinquini merged commit d7b9f3a into develop Mar 19, 2025
4 checks passed
@LucaCinquini LucaCinquini deleted the scaling-up-sps-core branch March 19, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants