Expand CONTRIBUTING.md (#6718)

Author: giordano

CONTRIBUTING.md

@@ -18,6 +18,13 @@ If you are submitting a recipe for a new package, follow the suggestions in the
 If you want to build a new version of an existing package, you can open a pull request which updates the sources and the version number of the package as appropriate.
 In most cases it is sufficient to change only these two properties of the recipe, but sometimes more work will be required if there are any changes in the build system (e.g. changed dependencies, or options for the build system) or bugs in the source code to deal with.
 
+### Compatibility tips
+
+When building binary packages we have to deal with [several incompatibilities](https://docs.binarybuilder.org/stable/tricksy_gotchas/).
+As a general remark, when using GCC as compiler (which is default when targeting Linux and Windows platforms) try to use the ***oldest*** versions that is able to compile your code, which can be selected with the `preferred_gcc_version` keyword argument to the [`build_tarballs`](https://docs.binarybuilder.org/stable/reference/#BinaryBuilder.build_tarballs) function, especially for [C++ code](https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html), this is the only way to get maximum compatibility.
+In any case, avoid using GCC 11+ when building C++, as that would be incompatible with Julia v1.6, the current Long Term Support version.
+There are currently no such problems with LLVM (the default compiler framework used when targeting macOS and FreeBSD), so that you can generally use the latest version of LLVM available, which is already the default.
+
 ### Recommendations and tips about commit messages
 
 Yggdrasil is a collection of recipes for hundreds of different packages.
@@ -43,26 +50,40 @@ are not descriptive (virtually all pull requests update a file called `build_tar
 
 #### Special keywords in commit messages
 
-There are some special tags that can be used anywhere in a commit message to do something special.
-Yggdrasil currently uses Azure Pipelines for CI, this allows using some [special keywords](https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#skipping-ci-for-individual-pushes) to allow skipping CI on commits, like `[skip ci]` or `[skip azp]`.
-However these have effect only if the keywords appear in the commit message of the tip of branch which triggered the workflow: in pull requests the tip of the branch is always the merge commit automatically generated by GitHub, so that `[skip ci]` in your commit will be ignored when running CI for a pull request.
-Instead, this does have effect if the keyword is included in the commit which triggers CI on the default branch, for example if the pull request has been squash-merged by a maintainer.
-
-Another special keyword which is `[skip build]`.  This is a two-fold effect:
+Anywhere in a commit message you can use the special keyword `[skip build]`, which has a two-fold effect:
 
 * in pull requests, a build of the touched packages will not be performed
 * for commits pushed to the default branch, a new build of the touched packages will not be performed but a new version of the correspoding JLL packages will be published, using as artifacts those from the latest registered versions of the packages (the build number will be increased by one).
 
+### Understanding build cache on Yggdrasil
+
+On Yggdrasil we run hundreds of builds every day.
+To minimise build times, we have a complex building infrastructure to cache as many build artifacts as possible:
+
+* BinaryBuilder uses by default [`ccache`](https://ccache.dev/) to cache compilers artifacts where possible;
+* on Yggdrasil we cache all successful builds, to avoid rebuilding packages unneedlessly.
+  This cache in indexed by:
+
+  * the git tree hash of the directory where the `build_tarballs.jl` file is
+  * the hash of the [`.ci/Manifest.toml`](.ci/Manifest.toml) file.
+
+This means that if you trigger a build which changes neither of them after a successful run, then a build will not actually happen.
+For this reason, please ***refrain from running builds which only rebase/merge your branch on the default branch*** after a fully successful build: there will be no build log, and reviewers will have a much harder time than necessary to find the audit logs.
+These rebuilds are also completely useless, as the build artifacts will in most cases be [bit-by-bit identical](https://docs.binarybuilder.org/stable/#Reproducibility) to the previous ones, only causing waste of time, resources, and energy.
+
 ## Information for maintainers
 
 Here are some recommendations for Yggdrasil maintainers:
 
 * before merging a pull request, make sure the log of the build does not contain important warnings or error messages such as
   * missing license file.  This is reported as an error in CI log during audit but it is not as a fatal error which would prevent successful build
   * "Linked library `<LIBRARY NAME>` could not be resolved and could not be auto-mapped"
-* make sure a pull request is not changing julia compat bounds without changing the version number of the package.
+* make sure a pull request is not changing julia compat bounds without changing the version number of the package
   This is particularly important when building an existing package for the first time for [aarch64-darwin or armv6l-linux](https://github.com/JuliaPackaging/Yggdrasil/issues/2763)
 * make sure dependencies of [packages with known incompatibilities](https://github.com/JuliaPackaging/Yggdrasil/issues/3024) are pinned correctly
 * always [squash and merge](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges#squash-and-merge-your-commits) pull requests, to keep history short and linear.
   There is little need to preserve full history of individual pull requests.
-  This also makes sure special keywords in commit messages are part of the commit which triggers the CI workflow and so they can have effect.
+  This also makes sure special keywords in commit messages are part of the commit which triggers the CI workflow and so they can have effect
+* use special commit message keywords as appropriate:
+  * `[skip ci]` to not run CI at all when merging a pull request
+  * `[skip build]` to only update the JLL wrapper of a package without rebuilding the package once again

README.md

@@ -1,6 +1,6 @@
 # Yggdrasil
 
-Build Status: [![Build Status](https://dev.azure.com/JuliaPackaging/Yggdrasil/_apis/build/status/JuliaPackaging.Yggdrasil?branchName=master)](https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/latest?definitionId=1&branchName=master)
+Build Status: [![Build status](https://badge.buildkite.com/20f068d74db24be50d79ac4710defa74e19d5d912e31f9bda2.svg)](https://buildkite.com/julialang/yggdrasil)
 
 ![Yggdrasil](https://user-images.githubusercontent.com/1282691/177174254-aa90664e-5c20-4ea3-9938-34de961dc198.png)
 
@@ -15,7 +15,7 @@ To contribute a new recipe, you can either
 
 Yggdrasil builds the tarballs using `master` version of BinaryBuilder.jl, which requires Julia 1.3.0 or later versions.  Note that this BinaryBuilder.jl version has some differences compared to v0.1.4 and the builders generated are slightly different.  You are welcome to contribute builders written for  BinaryBuilder.jl v0.1.4, but they will likely need minor adjustements.
 
-[Azure pipelines](https://dev.azure.com/JuliaPackaging/Yggdrasil/_build?view=runs) are used to test that the builders can successfully produce the tarballs.
+[Buildkite CI](https://buildkite.com/julialang/yggdrasil) is used to test that the builders can successfully produce the tarballs.
 
 If you prefer to test your manual buildscript before opening the pull request, we suggest installing `BinaryBuilder.jl` on Julia 1.3 or any following release and running `julia --color=yes build_tarballs.jl --verbose --debug` locally.  On MacOS, you will need to have `docker` installed for this to work.