Skip to content

Speed up CI builds with ccache (seeded from develop) #16356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
marcocastignoli wants to merge 7 commits into
base: develop
Choose a base branch
from
+79 −19
Draft

Speed up CI builds with ccache (seeded from develop) #16356

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f738513
Speed up CI builds with ccache
marcocastignoli Dec 17, 2025
51177ce
Address PR comments
marcocastignoli Dec 22, 2025
4df4fd0
Bumo versions of buildpack-deps dockerfiles
marcocastignoli Dec 22, 2025
456a48c
Skip ccache build when pushing tags
marcocastignoli Dec 23, 2025
321465f
Fix ccache permission error while testing buildpack-deps
marcocastignoli Dec 23, 2025
25d5d8b
Update docker images to use version with ccache
marcocastignoli Dec 23, 2025
b50f7d4
Fix cache key in restore_cache
marcocastignoli Dec 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 37 additions & 2 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,44 @@ commands:

run_build:
steps:
# TODO: remove the line below
# Before merging this PR we need to use an alternative develop branch to test the functionality, `develop-cache-test`
Comment on lines 209 to 212
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to be able to run without ccache on tags, so we still need to keep a non-cached variant of this command around.

Copy link
Member Author

@marcocastignoli marcocastignoli Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think is the best solution to handle this case? Something like this would work or do we need a more generic solution?

      - when:
          condition:
            and:
              - equal: ["", << pipeline.git.tag >>]
          steps:
            - run:
                name: Build (ccache)
                command: |
                  export CCACHE_DIR="$HOME/.ccache"
                  export CCACHE_BASEDIR="$(pwd)"
                  export CCACHE_NOHASHDIR=1
                  mkdir -p "$CCACHE_DIR"
                  export CMAKE_OPTIONS="${CMAKE_OPTIONS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
                  ccache -z

                  scripts/ci/build.sh

                  ccache -s
      - when:
          condition:
            or:
              - not:
                  equal: ["", << pipeline.git.tag >>]
          steps:
            - run:
                name: Build (no ccache)
                command: scripts/ci/build.sh

Copy link
Collaborator

@cameel cameel Dec 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that should work.

I mean, I would have preferred something that directly passes this decision as a parameter from the top, but I that would be annoying, because you'd have to modify most jobs to have parameters. A global version like this does not have this problem.


# On `develop-cache-test` we intentionally do NOT restore any ccache, so every run starts from scratch.
# Other branches restore the most recent cache produced on `develop-cache-test`.
- when:
condition:
not:
equal: ["develop-cache-test", << pipeline.git.branch >>]
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The breaking branch needs special treatment as well. It's not a feature branch but currently would be treated like one. It's like develop, but for the next breaking version.

steps:
- restore_cache:
keys:
# Reuse cache produced on develop-cache-test (prefix match; restores the most recent).
- v1-ccache-{{ arch }}-develop-cache-test-
- v1-ccache-{{ arch }}-
- run:
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we usually add a warning here because not everyone may realize that changing the code between restore_cache and save_cache doe not invalidate the cache automatically.

Suggested change
- v1-ccache-{{ arch }}-
- run:
- v1-ccache-{{ arch }}-
# WARNING! If you edit anything between here and save_cache, remember to bump version or invalidate the cache manually.
- run:

name: Build
command: scripts/ci/build.sh
command: |
export CCACHE_DIR="$HOME/.ccache"
export CCACHE_BASEDIR="$(pwd)"
export CCACHE_NOHASHDIR=1
mkdir -p "$CCACHE_DIR"
export CMAKE_OPTIONS="${CMAKE_OPTIONS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
ccache -z || true
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect not to have this on any of the platforms?
Is that platform Windows? :P


scripts/ci/build.sh

ccache -s || true

# On `develop-cache-test` we always generate a fresh cache by never restoring and by saving under a unique key.
- when:
condition:
equal: ["develop-cache-test", << pipeline.git.branch >>]
steps:
- save_cache:
key: v1-ccache-{{ arch }}-develop-cache-test-{{ .Revision }}
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect that we might want to invalidate cache whenever we suspect something weird is going on. It would be convenient to be able to do that without having to merge a PR. In an extreme case we might even not be able to merge one due to required jobs failing because of cache. In addition to the hard-coded version we could also keep a variable in CircleCI config that we can bump to get the same effect:

Suggested change
key: v1-ccache-{{ arch }}-develop-cache-test-{{ .Revision }}
key: v1-{{ .Environment.CCACHE_KEY }}-ccache-{{ arch }}-develop-cache-test-{{ .Revision }}

paths:
- ~/.ccache

run_build_ossfuzz:
steps:
Expand Down Expand Up @@ -1197,7 +1232,7 @@ jobs:
- run:
name: Install build dependencies
command: |
pacman --noconfirm -Syu --noprogressbar --needed base-devel boost cmake git openssh tar
pacman --noconfirm -Syu --noprogressbar --needed base-devel boost cmake ccache git openssh tar
- checkout
- run_build
- store_artifacts_solc
Expand Down
1 change: 1 addition & 0 deletions .circleci/osx_install_dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ then
brew update
brew upgrade
brew install cmake
brew install ccache
brew install wget
brew install coreutils
brew install diffutils
Expand Down
1 change: 1 addition & 0 deletions scripts/docker/buildpack-deps/Dockerfile.emscripten
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ RUN <<-EOF
apt-get update
apt-get install -qqy --no-install-recommends \
lsof \
ccache \
lz4 \
python3 \
python3-pip \
Expand Down
1 change: 1 addition & 0 deletions scripts/docker/buildpack-deps/Dockerfile.ubuntu.clang.ossfuzz
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ RUN apt-get update; \
automake \
bison \
build-essential \
ccache \
curl \
git \
jq \
Expand Down
1 change: 1 addition & 0 deletions scripts/docker/buildpack-deps/Dockerfile.ubuntu2404
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ RUN set -ex; \
apt-get install -qqy --no-install-recommends \
build-essential \
cmake \
ccache \
Copy link
Collaborator

@cameel cameel Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a special workflow for updating buildpack images and this needs to be done in two steps (separate PRs). See buildpack-deps/README.md.

jq \
libboost-filesystem-dev \
libboost-program-options-dev \
Expand Down
1 change: 1 addition & 0 deletions scripts/docker/buildpack-deps/Dockerfile.ubuntu2404.arm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ RUN <<-EOF
apt-get install --quiet=2 --no-install-recommends \
build-essential \
cmake \
ccache \
jq \
libboost-filesystem-dev \
libboost-program-options-dev \
Expand Down
1 change: 1 addition & 0 deletions scripts/docker/buildpack-deps/Dockerfile.ubuntu2404.clang
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ RUN set -ex; \
build-essential \
clang \
cmake \
ccache \
jq \
libboost-filesystem-dev \
libboost-program-options-dev \
Expand Down
Loading