ocamlbuild failing to build dependents if sandbox is not properly cleared up #13467
Description
Here is a reproduction case in docker:
FROM ocaml/opam:debian-12-ocaml-5.2
USER root
RUN apt-get update && apt-get install -y pkg-config
USER opam
WORKDIR /home/opam/test
COPY --chown=opam:opam dune-project dune main.ml ./
RUN opam install dune
RUN opam exec -- dune pkg lock
RUN opam exec -- dune build
# Create stale sandbox with empty library files
RUN OCAMLBUILD=$(find _build -name ocamlbuild -type f -executable | head -1) && \
STALE=$($OCAMLBUILD -where 2>&1 | grep "originally installed" | sed 's/.*(\([^)]*\)).*/\1/') && \
mkdir -p "$STALE" && \
touch "$STALE/ocamlbuild.cma" "$STALE/ocamlbuild.cmo" "$STALE/ocamlbuildlib.cma"
# Rebuild bytesrw - will fail with exit 10
RUN rm -rf _build/_private/default/.pkg/bytesrw* && \
opam exec -- dune build 2>&1; exit 0
We get the following:
$ docker run -it ocamlbuild-test bash
opam@70666478daa7:~/test$ opam exec -- dune build
File "dune.lock/bytesrw.0.3.0.pkg", line 8, characters 6-11:
8 | ocaml
^^^^^
Error: Logs for package bytesrw
pkg.ml: [ERROR] cmd ['ocamlbuild' '-use-ocamlfind' '-classic-display' '-j' '4' '-tag' 'debug'
'-build-dir' '_build' 'pkg.otarget']: exited with 10
The particular issue is really a problem of ocamlbuild. It hardcodes the sandbox path in which it was built. If that path doesn't exist, it uses a fallback which works hence why our overlay is working. However, if you have a partially cleared sandbox, which can happen when the file-system is slow or configured in a peculiar way like with docker, you may end up tricking ocamlbuild and causing it to fail.
The fix here is simple. Stop hardcoding such paths in ocamlbuild. Luckily, someone has already done all the hard work for us and it lives in David's relocatable compiler repository.
We should therefore merge the patches there with our overlay so that the relocatable ocamlbuild is picked instead. This should resolve the issue above even if the sandbox is not cleared correctly.
This is part of the work on