pkg: complex makefiles, cerberus-lib issue

[giltho]

Hello,

Bug description

I'm trying to build a project using dune package management instead of opam and I'm running about a specific issue involving cerberus-lib (with verbose logs):

Running[5]: (cd /Users/sacha/.cache/dune/git-repo && /usr/bin/git ls-remote https://github.com/giltho/cerberus) > /var/folders/zw/pnw89cw53s55ssszz27_0xh00000gn/T/dune_8d9e03_output
    Building cerberus-lib.dev
Running[7]: (cd _build/.sandbox/04dc67220b5a5c5b7d8ebf145397bd04/_private/default/.pkg/cerberus-lib.dev-a361fa1db4e4cd5ef05600950a8f25bd/source && /opt/homebrew/bin/gmake prelude-src) 2> /var/folders/zw/pnw89cw53s55ssszz27_0xh00000gn/T/dune-pkg_a8bdcf_stderr
Output[7]:
[MKDIR] ocaml_frontend/generated
[LEM] generating files in [ocaml_frontend/generated] (log in [ocaml_frontend/lem.log])
File "dune.lock/cerberus-lib.dev.pkg", line 8, characters 10-17:
8 |      (run %{make} prelude-src)
              ^^^^^^^
Error: Logs for package cerberus-lib
File "frontend/model/global.lem", line 1, character 1 to line 1, character 22
  Unknown dependency: could not find module 'Pervasives' in directories 'frontend/model', './library'
gmake: *** [Makefile:196: ocaml_frontend/generated/utils.ml] Error 1

The cerberus-lib build is non-standard and involves a makefile for building ocaml files using lem.

Repro

The cerberus repo is quite big so I've tried to minimise it for the needs of this repro, in this branch: https://github.com/giltho/cerberus/tree/minimize-for-repro

Looking at the opam file, the build is in two steps:

make prelude-src
dune build -p cerberus-lib @install

Running these two commands in a row from within the cerberus repository seem to work out just fine. However, running it from another repository that has pinned cerberus using dune package management fails.

I have also setup a reproduction repository at:
https://github.com/giltho/dune-repro-cerberus

Where, supposedly, dune pkg lock && dune build should reproduce the issue.

Additional info

• Macbook Pro M4 Max
• Sequoia 15.7.3
• dune 3.21.0
• OCaml 5.4.0

[Alizter]

There are a few issues that need addressing here:

1. Lem hardcodes a %{prefix}% in its build which becomes invalid due to appearing in a sandbox which is later destroyed.
2. This can be fixed by adding setenv: [ LEMBLIB = "%{share}%/library" ], triggering lem's fallback, to the opam file, however that creates a new issue. Dune does not currently expand opam variables in setenv which is a bug.
3. Such variables must be expanded a little bit more absolutely than the current default of a relative path.
4. Once we do this in Dune, another bug rears its head, and that is that we never expanded the stubdirs variable correctly from the compiler package before. This needs fixing.

I have reproductions and fixes for each of these issues. That will require some time to polish up and PR. Some of the work is related to #10294.

[giltho]

One viable option is also to make the prefix path parametrisable in lem
@kmemarian ?

[giltho]

Though it also looks like this helped identify two bugs in dune so that's also good hahaha