Artifacts with ENABLE_EXTENSIONS rebuild every run: predicted hooks-hash (-H) ≠ built hooks-hash

[igorpecovnik]

Summary

Any artifact whose board sets ENABLE_EXTENSIONS rebuilds on every matrix run and never caches. The artifact version's -H component (hash_hooks_and_functions) computed by the info-gatherer (compile.sh gha-matrix) differs from the one computed by the actual artifact build (compile.sh artifact) — at the same framework commit. So the build pushes one tag and matrix_prep forever checks for a different one → eternal cache miss.

Evidence (uboot-bananapim7-vendor, BRANCH=vendor, ENABLE_EXTENSIONS=v4l2loopback-dkms)

At armbian/build main 9a6ea452:

phase	-H value	OCI state
gha-matrix predicts (matrix job name / OCI check)	H40c4	ghcr.io/armbian/os/uboot-bananapim7-vendor:…-H40c4-… → 404
actual build produces + pushes (ORAS)	H8240	…-H8240-… → 200

The matrix predicts H40c4, checks OCI, gets 404, schedules a build. The build computes H8240, builds, pushes …-H8240-…. Next run: matrix predicts H40c4 again → 404 → rebuild. Forever.

Common factor

Only boards with ENABLE_EXTENSIONS are affected (e.g. bananapim7,vendor → v4l2loopback-dkms,mesa-vpu; rock-5c,vendor → mesa-vpu; the rk35xx vendor family generally). Boards with no extensions cache fine — their hooks set is empty so both phases agree.

Root cause

-H = sha256(hash_hooks + hash_uboot_functions) (artifact-uboot.sh#L85-L97). hash_hooks comes from dump_extension_method_sources_functions, which emits the bodies of every <hook>__<impl> extension function currently defined in the bash process (compgen -A function, extensions.sh#L99).

The extension manager initializes once per process and refuses further loads (extensions.sh#L140-L151):

[[ ${initialize_extension_manager_counter} -lt 1 ]] && [[ "${ENABLE_EXTENSIONS}" != "" ]] && { … enable_extension … }
# "no more extensions are allowed to load after this."

So a clean per-board artifact build defines exactly that board's extension hooks → H8240, but the info-gatherer computes versions for many artifacts in a context whose enabled-extension set does not match each artifact's real build set → H40c4. Hence the predicted hooks-hash ≠ the built one for every extension-bearing artifact. (kernel/rootfs/bsp artifacts hash hooks the same way and are likely affected too.)

Impact

Production is presumably silently rebuilding all extension-bearing artifacts every cycle (wasted runner time + churned OCI tags), in addition to the cache never warming for downstreams.

Proposed fix direction

Compute each artifact's hash_hooks against the exact extension set its build will use, during info-gathering — i.e. derive the hooks-hash per artifact from its own ENABLE_EXTENSIONS (isolated extension state), rather than from whatever the info-gather process happened to load. Equivalently: the info-gather must reproduce, per artifact, the same defined <hook>__<impl> set the standalone build produces.

Repro

./compile.sh gha-matrix … TARGETS_FILTER_INCLUDE=BOARD:bananapim7 → note predicted uboot-bananapim7-vendor=…-H<a>-…. Then run the produced artifact invocation → note pushed …-H<b>-…. <a> != <b>.

[igorpecovnik]

Correction to the mechanism (not cross-artifact accumulation)

I traced the info path further and the original "extension manager accumulates across artifacts" explanation is wrong — gather_json_output_from_armbian forks a fresh subprocess per artifact (ProcessPoolExecutor, armbian_utils.py#L536). So each artifact's version is computed in isolation.

The divergence is between the two modes of that isolated process:

	command	mode
matrix predicts H40c4	artifact-config-dump-json	CONFIG_DEFS_ONLY=yes
build produces H8240	artifact	full build

Both run do_main_configuration → initialize_extension_manager with the same ENABLE_EXTENSIONS, so the difference is that under CONFIG_DEFS_ONLY=yes a different set of <hook>__<impl> functions ends up defined than in the full build (an extension_prepare_config/extension_finish_config hook registers hook implementations conditionally on config that the defs-only path doesn't fully compute). dump_extension_method_sources_functions (compgen -A function) then hashes a different set → different -H.

Concrete next step to pin it

For one affected board, dump the registered hook set in both modes and diff:

# info/prediction mode
CONFIG_DEFS_ONLY=yes ./compile.sh artifact-config-dump-json WHAT=uboot BOARD=bananapim7 BRANCH=vendor \
  ENABLE_EXTENSIONS=v4l2loopback-dkms RELEASE=forky BUILD_MINIMAL=yes BUILD_DESKTOP=no BETA=yes
# vs full build's artifact_prepare_version

…then compare the dump_extension_method_sources_functions output for the uboot hook points. The delta is the hook(s) registered in the build but not in the defs-only dump (or vice-versa) — that's the thing to make consistent across both modes.

Candidate fixes (once the delta is known)

1. Ensure the defs-only path runs the same hook-registration steps as the full build (most correct, smallest behavioral change).
2. Or make hash_hooks derive from the enabled extension set + their source files rather than the runtime-defined function set, so it's identical regardless of mode.

I'm not opening a code PR yet: this is core artifact caching and a wrong change breaks it org-wide, and the fix can't be validated without a matrix+build run. The diff above is the gating step.

[rpardini]

There must be something else.

See:

PRE_PREPARED_HOST=yes CONFIG_DEFS_ONLY=yes ./compile.sh artifact-config-dump-json WHAT=uboot BOARD=bananapim7 BRANCH=vendor ENABLE_EXTENSIONS=v4l2loopback-dkms RELEASE=forky BUILD_MINIMAL=yes BUILD_DESKTOP=no BETA=yes PRE_PREPARED_HOST=yes CONFIG_DEFS_ONLY=yes DEBUG=yes

Produces exactly the same hashes and json as

PRE_PREPARED_HOST=yes CONFIG_DEFS_ONLY=yes ./compile.sh artifact-config-dump-json WHAT=uboot BOARD=bananapim7 BRANCH=vendor RELEASE=forky BUILD_MINIMAL=yes BUILD_DESKTOP=no BETA=yes PRE_PREPARED_HOST=yes CONFIG_DEFS_ONLY=yes DEBUG=yes

[rpardini]

Debugging this. ENABLE_EXTENSIONS=v4l2loopback-dkms seems to have nothing to do with it -- simple u-boot BOARD=bananapim7 BRANCH=vendor resolves to H40c4 in the prepare/matrix, and to H8240 when built.