The metadata is not deterministic error

[igiona]

We make use of this recipe to build our rust tools in yocto.
So far everything worked fine.

Our toolchain builds 2 flavors of the application, one in with some developer tools and one not.

We are now experiencing the following error on the second build in the toolchain (the first image builds correctly):

ERROR: When reparsing /github/workspace/meta-layers/poky/../my-meta/recipes-support/my-rust-tool/my-rust-tool_0.1.1.bb:do_compile, the basehash value changed from 67e8cb1e2e18e916e5e2918fcefec005ee0488cf5ab89da5682661da51eb9ede to 66cb7c6a2637b52191a7df3bd118dd7bb11d969925332c788e9ea197a19471d1. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake my-rust-tool -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake my-rust-tool -cdo_compile -Sprintdiff

We're not sure if this recipe is somehow responsible for this error. We tried to revert back to a known working version of the recipe but it also didn't work. So likely it's not the root cause of the problem.

Nonetheless, did someone experience such error when working with this recipe?
Google did not help the error appears a completely different scenarios and problems...

Cheers

[nastevens]

@igiona Could you please share the recipe you're using for your build? Feel free to censor any sensitive/proprietary information. While this probably isn't an issue in meta-rust-bin itself, it's always possible that it's interacting with your recipe in an unexpected way.

[igiona]

@igiona Could you please share the recipe you're using for your build? Feel free to censor any sensitive/proprietary information. While this probably isn't an issue in meta-rust-bin itself, it's always possible that it's interacting with your recipe in an unexpected way.

Thank you for you answer @nastevens, appreciated!
Sure, here is the list of the files for the recipe, there also the files for the runit start up of the daemon, but given the their static-content nature I doubt that are related hence I'm not reporting them here:

my-tool_0.1.0.bb

require my-tool.inc

SRCREV = "<the git hash>"

my-tool_git.bb

require my-tool.inc

SRCREV = "${AUTOREV}"
PV = "git-${SRCPV}"

my-tool.inc

SUMMARY = "My tool"
AUTHOR = "ME"
HOMEPAGE = "https://me.com/"
LICENSE = "MY-1.0"
LIC_FILES_CHKSUM = "file://${MY_LIC_PATH};md5=${MY_LIC_CHECKSUM}"

inherit cargo_bin update-rc.d useradd runit-services

DEPENDS = " \
  "

RDEPENDS_${PN} = " \
  another-tool \
  "

BRANCH ?= "main"

SRC_URI = " \
  gitsm://[email protected]/MyOrg/my-tool.git;protocol=ssh;branch=${BRANCH} \
  file://runit/my-tool \
  file://runit/my-tool.check \
  "

S = "${WORKDIR}/git"

USERADD_PACKAGES = "${PN}"
USERADD_PARAM_${PN} = "--system --home /dev/null --shell /bin/false ${BPN}"
# Add to the necessary groups
GROUPADD_PARAM_${PN} = "--system my-clients ; --system mymy-clients"
GROUPMEMS_PARAM_${PN} = "-a ${BPN} -g my-clients ; -a ${BPN} -g mymy-clients"

INITSCRIPT_NAME = "${BPN}"
INITSCRIPT_PARAMS = "defaults 99"

RUNIT_SERVICES = "my-tool.default"

EXTRA_CARGO_FLAGS = "--config env.A_KEY=\"${A_KEY}\""
EXTRA_CARGO_FLAGS += " --config env.B_KEY=\"${B_KEY}\""
EXTRA_CARGO_FLAGS += " --config env.C_KEY=\"${C_KEY}\""
EXTRA_CARGO_FLAGS += " --config env.D_KEY=\"${D_KEY}\""

do_install_append() {
  # install the config file

  MY_FILES_ROOT=${D}${localstatedir}/lib/${BPN}/.myfolder
  MY_FILE_FILE_PATH=${MY_FILES_ROOT}/config

  install -d -m 0750 ${MY_FILES_ROOT}

  echo "settings" >> ${MY_FILE_FILE_PATH} && \
  echo "var1=${SOME_VAR1}" >> ${MY_FILE_FILE_PATH} && \
  echo "var2=${SOME_VAR2}" >> ${MY_FILE_FILE_PATH}
}

The various env variables are provide (x_KEY and SOME_VARy) are provided by the GitHub action and are not changing in the pipeline run.

[nastevens]

Thanks @igiona! The error you're getting happens when a variable changes from one parse of the recipe to the next - this most commonly happens when a timestamp is included somewhere in a variable or bitbake step. How is the GitHub Action providing the values of x_KEY/SOME_VARy? I ask because environment variables aren't automatically included into Yocto recipes, so those variables could be getting generated including dynamic Python functions.

You can actually find out the variable that's changing between runs by using the bitbake-diffsigs tool. There's a short article on it on the Yocto Wiki. In your case I think you should be able to do:

$ bitbake -S none my-rust-tool
$ bitbake -S printdiff my-rust-tool

That should show you what's changing on each execution and hopefully point to something dynamic. Like I said above, look particularly for changing dates/times/unix timestamps. Next steps will be figuring out where to fix it or to ignore it - let's figure out what's changing and then we can address that.

[igiona]

@nastevens thank you for the suggestion.

I tried that but didn't return much of useful information to me...

$ MACHINE=${BUILD_MACHINE} DISTRO=${BUILD_DISTRO}
bitbake -S none my-tool

NOTE: Started PRServer with DBfile: /github/workspace/build-docker/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 40167, PID: 3081014
Loading cache...done.
Loaded 4071 entries from dependency cache.
Parsing recipes...done.
Parsing of 2757 .bb files complete (2722 cached, 35 parsed). 4131 targets, 305 skipped, 0 masked, 0 errors.
WARNING: No recipes in default available for:
  /github/workspace/meta-layers/poky/../meta-chip-vendor/recipes-kernel/linux/linux-XYZ%.bbappend
  /github/workspace/meta-layers/poky/../meta-chip-vendor/recipes-multimedia/gstreamer/gstreamerXYZ.bbappend
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.50.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-myorg-linux-gnueabi"
MACHINE              = "myorg-product"
DISTRO               = "myorg-linux"
DISTRO_VERSION       = "X.Y.Z"
TUNE_FEATURES        = "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU           = "hard"
meta                 
meta-poky            
meta-yocto-bsp       = "HEAD:1cefabe0f0a2dd2ca8b6eb83e8e0824b72da941a"
meta-oe              
meta-python          
meta-networking      
meta-webserver       = "HEAD:786d3aa313e31a718acc9611ba3adb60573ec458"
meta-swupdate        = "HEAD:85dd34243d92a33a6ff9215245565f3a5dd4cc99"
meta-freescale       = "HEAD:f0be684f01b53482cb43e016a5c5c1faf3ae448e"
meta-freescale-3rdparty = "HEAD:261039773003cac16afbb497d800ca9ad8c6e5b8"
meta-qt5             = "HEAD:bbdf387d2b2db9cddf571b322448221d244c8021"
meta-rust-bin        = "HEAD:019e3b0073510e6f39fac23d9a3c2dd6d793a2fe"
meta-raspberrypi     = "HEAD:b4ec97e4eb8e36efd1f7e2f8ae020a9e55cfc239"
meta-chip-vendor-bsp-common = "HEAD:5e1ad1e63a85287656bb8bd41c85e1979bc2c03c"
meta-chip-vendor     = "HEAD:3112c5877d4dba188126062998de65501f6e894b"
meta-myorg           = "HEAD:e34034da2e89d58f017944bc34ed77abac76cf74"
meta-myorg-nonfree   = "HEAD:6e13ea3a32592c5ec622956043b5153bd51e5782"

Initialising tasks...done.
NOTE: Reparsing files to collect dependency data
ERROR: When reparsing /github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile, the basehash value changed from 98e85a2f0a455a467c8c9b3c9712e087ba8fc09780008849a5fdeae9f688ede0 to 30e87e6a607382b8ed7ad5a64541f02925f9c3a8e1d8e9b3ce51b974509e7f16. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake my-tool -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake my-tool -cdo_compile -Sprintdiff

ERROR: Bitbake's cached basehash does not match the one we just generated (/github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile)!
ERROR: The mismatched hashes were 98e85a2f0a455a467c8c9b3c9712e087ba8fc09780008849a5fdeae9f688ede0 and 30e87e6a607382b8ed7ad5a64541f02925f9c3a8e1d8e9b3ce51b974509e7f16
ERROR: Taskhash mismatch 897349a96a00dfd31bcfd06b9b28a225797fa553bb15b09cd3c044453acd02c7 versus 7d8c5d85cb16a95d2b0a445a4cf1163d0b008d054fb5354ab6c0ff1798ae89d1 for /github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile
Writing locked sigs to /github/workspace/build-docker/locked-sigs.inc

Summary: There was 1 WARNING message shown.
Summary: There were 8 ERROR messages shown, returning a non-zero exit code.

$ MACHINE=${BUILD_MACHINE} DISTRO=${BUILD_DISTRO}
bitbake -S printdiff my-tool

NOTE: Started PRServer with DBfile: /github/workspace/build-docker/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 33487, PID: 3081395
Loading cache...done.
Loaded 4071 entries from dependency cache.
Parsing recipes...done.
Parsing of 2757 .bb files complete (2722 cached, 35 parsed). 4131 targets, 305 skipped, 0 masked, 0 errors.
WARNING: No recipes in default available for:
  /github/workspace/meta-layers/poky/../meta-chip-vendor/recipes-kernel/linux/linux-XYZ%.bbappend
  /github/workspace/meta-layers/poky/../meta-chip-vendor/recipes-multimedia/gstreamer/gstreamerXYZ.bbappend
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.50.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-myorg-linux-gnueabi"
MACHINE              = "myorg-product"
DISTRO               = "myorg-linux"
DISTRO_VERSION       = "X.Y.Z"
TUNE_FEATURES        = "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU           = "hard"
meta                 
meta-poky            
meta-yocto-bsp       = "HEAD:1cefabe0f0a2dd2ca8b6eb83e8e0824b72da941a"
meta-oe              
meta-python          
meta-networking      
meta-webserver       = "HEAD:786d3aa313e31a718acc9611ba3adb60573ec458"
meta-swupdate        = "HEAD:85dd34243d92a33a6ff9215245565f3a5dd4cc99"
meta-freescale       = "HEAD:f0be684f01b53482cb43e016a5c5c1faf3ae448e"
meta-freescale-3rdparty = "HEAD:261039773003cac16afbb497d800ca9ad8c6e5b8"
meta-qt5             = "HEAD:bbdf387d2b2db9cddf571b322448221d244c8021"
meta-rust-bin        = "HEAD:019e3b0073510e6f39fac23d9a3c2dd6d793a2fe"
meta-raspberrypi     = "HEAD:b4ec97e4eb8e36efd1f7e2f8ae020a9e55cfc239"
meta-chip-vendor-bsp-common = "HEAD:5e1ad1e63a85287656bb8bd41c85e1979bc2c03c"
meta-chip-vendor     = "HEAD:3112c5877d4dba188126062998de65501f6e894b"
meta-myorg           = "HEAD:e34034da2e89d58f017944bc34ed77abac76cf74"
meta-myorg-nonfree   = "HEAD:6e13ea3a32592c5ec622956043b5153bd51e5782"

Initialising tasks...done.
The differences between the current build and any cached tasks start at the following tasks:
/github/workspace/meta-layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_install
NOTE: Reparsing files to collect dependency data
ERROR: When reparsing /github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile, the basehash value changed from 98e85a2f0a455a467c8c9b3c9712e087ba8fc09780008849a5fdeae9f688ede0 to 30e87e6a607382b8ed7ad5a64541f02925f9c3a8e1d8e9b3ce51b974509e7f16. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake my-tool -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake my-tool -cdo_compile -Sprintdiff

ERROR: Bitbake's cached basehash does not match the one we just generated (/github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile)!
ERROR: The mismatched hashes were 98e85a2f0a455a467c8c9b3c9712e087ba8fc09780008849a5fdeae9f688ede0 and 30e87e6a607382b8ed7ad5a64541f02925f9c3a8e1d8e9b3ce51b974509e7f16
ERROR: Taskhash mismatch 897349a96a00dfd31bcfd06b9b28a225797fa553bb15b09cd3c044453acd02c7 versus 7d8c5d85cb16a95d2b0a445a4cf1163d0b008d054fb5354ab6c0ff1798ae89d1 for /github/workspace/meta-layers/poky/../meta-myorg-nonfree/recipes-support/my-tool/my-tool_0.2.1.bb:do_compile
Writing locked sigs to /github/workspace/build-docker/locked-sigs.inc

Task base-files:do_install couldn't be used from the cache because:
  We need hash ec4ed9365b25554b0cddf231dfd06228a228ba8c319057bc83c46d6fb4dca960, closest matching task was c46fb6fc4bb7beb56025cfedcf7ce534e44706926fe095345b6b029a188296c0
  Taint (by forced/invalidated task) changed from nostamp(uuid4):dca1d824-eacb-4e7e-aa50-329783508cf7 to nostamp(uuid4):9a39f094-e9e2-408b-889e-7fea20d9845b

Summary: There was 1 WARNING message shown.
Summary: There were 8 ERROR messages shown, returning a non-zero exit code.

[nastevens]

@igiona In this case the error message suggestion is on the right track - you'll need to stop the hash generation at the do_compile step to avoid the whole build failing out. So the commands should be:

$ bitbake -cdo_compile -Snone my-tool
$ bitbake -cdo_compile -Sprintdiff my-tool

If that doesn't work you can use the bitbake-diffsigs tool manually as well with the *.sigdata.* files located in tmp/stamps/<your arch>/my-tool. Try ls *do_compile.sigdata.* in that directory and there should be more than one option, which you can then diff with bitbake-diffsigs <a> <b>.

[igiona]

bitbake-diffsigs

@nastevens thanks! We nailed it!
...and as suspected, it wasn't this recipe causing the issue 😉

This command:
bitbake-diffsigs -t my-tool do_compile`

pointed to the problem nicely:

NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /github/workspace/build-docker/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 38533, PID: 3081315
basehash changed from 3b8b865d202368b4b85d92d07622314abb0e8d73db9e0bb6c9911d2b04af8351 to b0308c98a3a5cdf286c95657e30a1fd84646c556739cd988473eafcac33d8002
Variable SSH_AUTH_SOCK value changed from '/tmp/ssh-AA3M49nf2qUY/agent.947' to '/tmp/ssh-3khez8gdXkw1/agent.2977323'

We indeed added this command before every recipe build, in order to allow cargo to access our crates hosted on our repos and accessed via SSH:

eval "$(ssh-agent -s)"

Now we start the agent only one time before running the recipes, this solves our issue!
...if someone would have told me this, I wouldn't have believed it 😵 😲

Are you aware of a way of tell bitbake that the SSH variable is not likely to change the build reproducibility and hence not to consider it?

Thanks again!

[nastevens]

@igiona Yup, you can work around this by adding BB_BASEHASH_IGNORE_VARS += "SSH_AUTH_LOCK" in a top-level config like local.conf or your machine conf.

While looking into this I realized that the root cause of this problem is kind of in meta-rust-bin - we currently don't split the fetch step from the build step for cargo like Yocto wants us to. Bitbake actually has an exception for SSH_AUTH_SOCK when used in do_fetch, but we do downloads in the do_compile step. See #150 - I believe once this behavior is changed you'll be able to remove the workaround. Needing SSH is also a very common use case so it'd probably be good for us to update the documentation to better cover it.

Thanks for opening the issue!

[igiona]

@nastevens interesting point, and I'm glad that our struggle at least will make this recipe better 😄

Would it be possible to enable start the ssh agent on the recipe itself?
To me, it was cumbersome to have to do it in the first place.
Is perhaps started in the do_fetch step by bitbake itself?

Do you have a ETA for #150 ?

[nastevens]

@igiona

Would it be possible to enable start the ssh agent on the recipe itself?

Interesting idea - would you mind opening a new issue with this as a feature request? I don't think enabling by default would be a good idea but I could definitely see including it as an optional thing, maybe as a separate class/recipe.

Do you have a ETA for #150 ?

I currently work on this project in my free time so giving ETAs for anything is tough. Knowing that someone wants to use the feature though is an excellent way to move things up the list!

[igiona]

@igiona

Would it be possible to enable start the ssh agent on the recipe itself?

Interesting idea - would you mind opening a new issue with this as a feature request? I don't think enabling by default would be a good idea but I could definitely see including it as an optional thing, maybe as a separate class/recipe.

Sure, here you are: #168

Do you have a ETA for #150 ?

I currently work on this project in my free time so giving ETAs for anything is tough. Knowing that someone wants to use the feature though is an excellent way to move things up the list!

No worries, I totally understand 👍
Many thanks for your efforts on maintaining this recipe!