Refactor build pipeline to reduce duplicate builds

[leighmcculloch]

What

Refactor build pipeline making it build all the dependency software first for all images, deduplicating them so that it builds each dependency once, then build the quickstart images.

The new workflow also comes with some bonus features:

• It can run scheduled builds, such as a nightly-like image build.
• It can be triggered via a workflow call from another repository, that allows another repository to build any quickstart with any configuration of software, and to run quickstarts standard "can everything start up and sync" tests. That resulting image can be used in a subsequent test.
• It can accept branch references for software components and handle them stably across the build pipeline.

Old Workflow

flowchart TD
    subgraph Main Branch Workflow
        subgraph Latest Pipeline
            subgraph Latest AMD64 Deps Pipeline
                B1_latest_amd64[xdr] --> B_latest_amd64
                B2_latest_amd64[core] --> B_latest_amd64
                B3_latest_amd64[rpc] --> B_latest_amd64
                B4_latest_amd64[horizon] --> B_latest_amd64
                B5_latest_amd64[friendbot] --> B_latest_amd64
                B6_latest_amd64[lab] --> B_latest_amd64
            end

            B_latest_amd64[load/prepare/build deps amd64] --> C_latest_amd64[build latest quickstart amd64]
            C_latest_amd64 --> D_latest_amd64[test latest amd64]
            D_latest_amd64 --> E_latest_amd64[push latest amd64]
            E_latest_amd64 --> F_latest[push-manifest latest amd64+arm64]
            F_latest --> G_latest[action latest amd64+arm64]

            subgraph Latest ARM64 Deps Pipeline
                B1_latest_arm64[xdr] --> B_latest_arm64
                B2_latest_arm64[core] --> B_latest_arm64
                B3_latest_arm64[rpc] --> B_latest_arm64
                B4_latest_arm64[horizon] --> B_latest_arm64
                B5_latest_arm64[friendbot] --> B_latest_arm64
                B6_latest_arm64[lab] --> B_latest_arm64
            end

            B_latest_arm64[load/prepare/build deps arm64] --> C_latest_arm64[build latest quickstart arm64]
            C_latest_arm64 --> D_latest_arm64[test latest arm64]
            D_latest_arm64 --> E_latest_arm64[push latest arm64]
            E_latest_arm64 --> F_latest
        end

        subgraph Testing Pipeline
            subgraph Testing AMD64 Deps Pipeline
                B1_testing_amd64[xdr] --> B_testing_amd64
                B2_testing_amd64[core] --> B_testing_amd64
                B3_testing_amd64[rpc] --> B_testing_amd64
                B4_testing_amd64[horizon] --> B_testing_amd64
                B5_testing_amd64[friendbot] --> B_testing_amd64
                B6_testing_amd64[lab] --> B_testing_amd64
            end

            B_testing_amd64[load/prepare/build deps amd64] --> C_testing_amd64[build testing quickstart amd64]
            C_testing_amd64 --> D_testing_amd64[test testing amd64]
            D_testing_amd64 --> E_testing_amd64[push testing amd64]
            E_testing_amd64 --> F_testing[push-manifest testing amd64+arm64]
            F_testing --> G_testing[action testing amd64+arm64]

            subgraph Testing ARM64 Deps Pipeline
                B1_testing_arm64[xdr] --> B_testing_arm64
                B2_testing_arm64[core] --> B_testing_arm64
                B3_testing_arm64[rpc] --> B_testing_arm64
                B4_testing_arm64[horizon] --> B_testing_arm64
                B5_testing_arm64[friendbot] --> B_testing_arm64
                B6_testing_arm64[lab] --> B_testing_arm64
            end

            B_testing_arm64[load/prepare/build deps arm64] --> C_testing_arm64[build testing quickstart arm64]
            C_testing_arm64 --> D_testing_arm64[test testing arm64]
            D_testing_arm64 --> E_testing_arm64[push testing arm64]
            E_testing_arm64 --> F_testing
        end

        subgraph Future Pipeline
            subgraph Future AMD64 Deps Pipeline
                B1_future_amd64[xdr] --> B_future_amd64
                B2_future_amd64[core] --> B_future_amd64
                B3_future_amd64[rpc] --> B_future_amd64
                B4_future_amd64[horizon] --> B_future_amd64
                B5_future_amd64[friendbot] --> B_future_amd64
                B6_future_amd64[lab] --> B_future_amd64
            end

            B_future_amd64[load/prepare/build deps amd64] --> C_future_amd64[build future quickstart amd64]
            C_future_amd64 --> D_future_amd64[test future amd64]
            D_future_amd64 --> E_future_amd64[push future amd64]
            E_future_amd64 --> F_future[push-manifest future amd64+arm64]
            F_future --> G_future[action future amd64+arm64]

            subgraph Future ARM64 Deps Pipeline
                B1_future_arm64[xdr] --> B_future_arm64
                B2_future_arm64[core] --> B_future_arm64
                B3_future_arm64[rpc] --> B_future_arm64
                B4_future_arm64[horizon] --> B_future_arm64
                B5_future_arm64[friendbot] --> B_future_arm64
                B6_future_arm64[lab] --> B_future_arm64
            end

            B_future_arm64[load/prepare/build deps arm64] --> C_future_arm64[build future quickstart arm64]
            C_future_arm64 --> D_future_arm64[test future arm64]
            D_future_arm64 --> E_future_arm64[push future arm64]
            E_future_arm64 --> F_future
        end

        G_latest --> H[complete]
        G_testing --> H
        G_future --> H
    end

Loading

New Workflow

flowchart TD
    subgraph New Workflow
        I[setup]

        subgraph Deps Pipeline
            I --> J1[xdr amd64]
            I --> J2[xdr arm64]
            I --> J3[core amd64]
            I --> J4[core arm64]
            I --> J5[rpc amd64]
            I --> J6[rpc arm64]
            I --> J7[horizon amd64]
            I --> J8[horizon arm64]
            I --> J9[friendbot amd64]
            I --> J10[friendbot arm64]
            I --> J11[lab amd64]
            I --> J12[lab arm64]
            J1 --> R
            J2 --> R
            J3 --> R
            J4 --> R
            J5 --> R
            J6 --> R
            J7 --> R
            J8 --> R
            J9 --> R
            J10 --> R
            J11 --> R
            J12 --> R
        end

        R[load/prepare/build deps]

        subgraph Latest Pipeline
            R --> L_latest_amd64[build latest quickstart amd64]
            R --> L_latest_arm64[build latest quickstart arm64]
            L_latest_amd64 --> M_latest_amd64[test latest amd64]
            L_latest_arm64 --> M_latest_arm64[test latest arm64]
            M_latest_amd64 --> N_latest_amd64[push latest amd64]
            M_latest_arm64 --> N_latest_arm64[push latest arm64]
            N_latest_amd64 --> O_latest[push-manifest latest amd64+arm64]
            N_latest_arm64 --> O_latest
            O_latest --> P_latest[action latest amd64+arm64]
        end

        subgraph Testing Pipeline
            R --> L_testing_amd64[build testing quickstart amd64]
            R --> L_testing_arm64[build testing quickstart arm64]
            L_testing_amd64 --> M_testing_amd64[test testing amd64]
            L_testing_arm64 --> M_testing_arm64[test testing arm64]
            M_testing_amd64 --> N_testing_amd64[push testing amd64]
            M_testing_arm64 --> N_testing_arm64[push testing arm64]
            N_testing_amd64 --> O_testing[push-manifest testing amd64+arm64]
            N_testing_arm64 --> O_testing
            O_testing --> P_testing[action testing amd64+arm64]
        end

        subgraph Future Pipeline
            R --> L_future_amd64[build future quickstart amd64]
            R --> L_future_arm64[build future quickstart arm64]
            L_future_amd64 --> M_future_amd64[test future amd64]
            L_future_arm64 --> M_future_arm64[test future arm64]
            M_future_amd64 --> N_future_amd64[push future amd64]
            M_future_arm64 --> N_future_arm64[push future arm64]
            N_future_amd64 --> O_future[push-manifest future amd64+arm64]
            N_future_arm64 --> O_future
            O_future --> P_future[action future amd64+arm64]
        end

        P_latest --> Q[complete]
        P_testing --> Q
        P_future --> Q
    end

Loading

Why

To make the build pipeline more scalable. To support adding more image types, such as nightly builds with and without vnext. Most of the time the various images of quickstart share the same internal software components, and adding new images blows out the build effort.

To make the build pipeline more useful. In the past we've built secondary pipelines in things like system-test, rather than leveraging the caching and build process that exists here in quickstart.

Close #754