Reduce GitHub Actions resource usage: size of artifacts, cpu time

[leighmcculloch]

Reduce GitHub Actions resource usage: size of artifacts, cpu time.

Quickstart builds a lot of components and the efficiency of those builds is degrading. There are things we can do to reduce the amount of work the build pipeline is doing without losing anything we have today.

A large impact to the current slowness of the build pipeline is that the pipeline generates large artifacts that get copied between jobs.

The build pipeline already does a heap of caching, which is working really well, and doesn't need to change.

Here's some stats to illustrate how this has changed over time. In recent times there have been more builds on the repo running from increased contributions. There were also three large images added to the pipeline for rpc, lab, and the stellar-xdr cli.

Sizes of individual images in build pipeline:

Note: large variance in cores build size over the last year has resulted in it showing up in this graph larger than it really is in recent months. The other images are pretty spot on and have not varied much.

Size of all images across a year:

Sizes of images in the different architectures:

Size of artifacts being written over time:

TODO:

• Improvements to the GitHub Action tests #747
• Add build caching for Horizon and Friendbot #748
• Fail faster on job failure or build cancellation #750
• Reduce size of horizon, lab, xdr images Reduce build time by reducing the size of intermediary build stages to necessary files only #749
• Set smaller retention period for intermediary images Specify retention days for CI workflow artifacts #755
• Remove free up disk space step that is no longer needed and costs cpu time Remove 'Free up disk space' step from GitHub workflow #756
• Reduce size of friendbot images Move Friendbot Dockerfile into repo and reduce size #757
• Reduce size of rpc images Move RPC Dockerfile into repo and reduce size #758
• Reduce size of core images Move Core Dockerfile into repo and reduce size #769
• Reduce need to rerun builds repeatedly to deal with failures Reduce test jobs that run for pubnet and testnet #774
• Switch to free arm runners Use free arm runners instead of custom runner #775
• Manually delete the backlog of 9TB of artifacts
• Share intermediary images across all final image builds Refactor build pipeline to reduce duplicate builds #778

[leighmcculloch]

Sizes of individual images in build pipeline after the changes that have been made so far in #747, #748, #749, #750: