Updating build to be reproducible

[bdemers]

Multiple builds (without changes) will produce the same output (e.g. the hash of the jars will be identical)

Note

This can be tested by running:

mvn clean install && mvn clean verify artifact:compare

[bdemers]

IIRC, this will fail CI on the older JVMs
PR is marked as draft as we likely want to wait a bit on this one

[lhazlewood]

I know this is probably counter to reproducibility, but you can set different versions for different JVMs in the <profiles> section at the bottom as a version property. Or have a different plugin configuration entirely. Just a thought.

[bdemers]

That could work 🤔
I've been wanting to try out a different idea too, building with a newer JDK, and using the toolchain option in the Surefire plugin to test with 7.
https://github.com/actions/setup-java/#maven-options

But, I have a feeling that's more work than it's worth, given that support for 7 will be dropped. 🤷

[lhazlewood]

Yeah, I dunno how much work you'd want to put in for JDK 7. We won't need it much longer.

[bdemers]

I updated using the profile technique used, it's not worth spending more time with 7 😄

[lhazlewood]

@bdemers do you want to try this again given all that stuff has been merged to master? If not, please close. 😄

[bdemers]

We might want to add a CI check in here 🤔
Running the build twice for the full (non-7) matrix is probably overkill...

Maybe pick a version, like the min supported reproducible version, (zulu-8)? and add another job?
What do you think?

[lhazlewood]

We might want to add a CI check in here 🤔 Running the build twice for the full (non-7) matrix is probably overkill...

Maybe pick a version, like the min supported reproducible version, (zulu-8)? and add another job? What do you think?

What do you mean run it twice? I'm not following 😅

[bdemers]

What do you mean run it twice? I'm not following 😅

Sorry about that!
To check of a project/build reproducible, the build is run twice and the output compared,

Basically this:

mvn clean install && mvn clean verify artifact:compare

This would double the run time, which I don't think is worth doing. We could run the above against a single target (potentially even skipping tests, to speed things up), I'm not sure we should worry about this at all in CI right now. It's something we could check periodically or at release time.

@hboutemy do you have any thoughts/recommendations for projects on how to validate reproducibility via CI?

[lhazlewood]

@bdemers why do you need to run the tests twice? Isn't the goal to ensure that the artifacts are identical? So you just need to run package and skip the tests entirely?

[hboutemy]

do you have any thoughts/recommendations for projects on how to validate reproducibility via CI?

I recommend NOT trying to validate reproduciblity via CI: there is no reason reproduciblity will change from commit to commit

checking reproducibility by hand is sufficient, and even necessary because 2 builds on the same environment may give the same result, but not on 2 different environments (for example if the current directory is put in the output)

then real checking requires ore than just normal CI

[hboutemy]

3.5.1 is known to have some RB issues: why not upgrade to 5.1.9?

[lhazlewood]

We still support JDK 7, so that's probably it.

[bdemers]

yeah 😭, when testing with Java 7 we use the older version, when testing with Java 8+ we use the latest.

We should probably use toolchains for testing with Java 7, but... it's getting dropped soon, so this is a temporary hack.

[bdemers]

I recommend NOT trying to validate reproduciblity via CI: there is no reason reproduciblity will change from commit to commit

checking reproducibility by hand is sufficient, and even necessary because 2 builds on the same environment may give the same result, but not on 2 different environments (for example if the current directory is put in the output)

then real checking requires ore than just normal CI

Thanks @hboutemy, great points!