Tools and rules for integrating opam (the OCaml package manager) onto Bazel for use by rules_ocaml.
Currently the only tool is the opam module extension. The Bazel module extension facility is designed to support integration of external
non-Bazel tools into the Bazel package management (bzlmod) and build
system. The tools_opam extension seamlessly integrates opam into bazel so that
rules_ocaml can depend on opam resources without any special configuration. You must tell the extension which switch you want and what packages you need, but the tool itself requires no configuration.
Install Bazel. Then:
Clone a demo
git clone https://github.com/obazl/demo_hello.git cd demo_hello bazel run bin:hello bazel test test bazel build //... (1) bazel test //... (2) bazel test //... --build_tests_only (3)
-
Builds all targets
-
Builds all targets and runs all tests
-
Runs all tests but only builds targets required by the tests
bazel_dep(name = "tools_opam", version = "1.0.0")
opam = use_extension("@tools_opam//extensions:opam.bzl", "opam")
opam.deps( (1)
toolchain = "xdg", # "global" | "local" | "xdg" (default)
opam_version = "2.3.0", # ignored unless toolchain = "xdg"
ocaml_version = "5.3.0",
pkgs = {"cmdliner": "1.3.0"} # version ignored currently
)
use_repo(opam, "opam.cmdliner") (2)
use_repo(opam, "opam", "opam.ocamlsdk") (3)
register_toolchains("@opam.ocamlsdk//toolchain/selectors/local:all") (4)
register_toolchains("@opam.ocamlsdk//toolchain/profiles:all") (4)
opamdev = use_extension("@tools_opam//extensions:opam.bzl", "opam", (5)
dev_dependencies = True)
opamdev.deps(pkgs = {"ounit2": "2.2.7"})
-
depsis a "tag" supported by the extension; a tag is essentially a method. This instruction tells Bazel to execute the module extension function; see below for more information. -
The module extension function runs a "repo rule" for every package in
pkgs(and all their dependencies); theuse_repodirective tells Bazel which of those repos are direct dependencies of this (root) module. It "imports" the module, making it accessible toBUILD.bazelfiles in this module. -
The
opamandopam.ocamlsdkmodules are always implicitly created by the extension and must be imported. Do not list inpkgs. -
The toolchains created by
opam.ocamlsdkmust be registered. -
The same extension used with
dev_dependencies = Truemeans it will be ignored if the current module is not the root module.
With legacy build systems (Dune, ocamlfind, etc.) the opam package name is sufficient to express a dependency in a build file. With Bazel, we need:
-
a label expressing a repo (e.g.
@opam.ounit2) -
a (Bazel) package within the repo (e.g.
//lib) -
a target within the package (e.g.
:ounit2)
For example, @opam.ounit2//lib:ounit2.
ocaml_module(name = "Test", struct = "test.ml",
deps = ["@opam.ounit2//lib", ...]) (1)
-
@opam.ounit2//libabbreviates@opam.ounit2//lib:lib, which is an alias of@opam.ounit2//lib:ounit2. Any of the three forms may be used.
|
Warning
|
Do not confuse Bazel’s concept of package with that of opam (or Dune, or findlib, or …). See the Bazel documentation on packages for details. |
The extension adds prefix opam. to all opam package names, which
means your build files will refer to them as @opam.pkg (in this
example @opam.ounit2). This makes it easy to distinguish between
opam and non-opam dependencies in your build files. If you prefer to
drop the prefix, or name your packages something else, you can use the
aliasing facility of use_repo:
use_repo(opam, ounit2 = "opam.ounit2") # use `@ounit2` in BUILD.bazel files
The mapping scheme is straightforward. A package pkg with no
subpackages becomes @opam.pkg//lib (or equivalently
@opam.pkg//lib:lib or @opam.pkg//lib:pkg). Subpackages become
Bazel packages within the repo; i.e. mtime.clock becomes
@opam.mtime//clock/lib (equivalently, @opam.mtime//clock/lib:lib, @opam.mtime//clock/lib:clock).
| opam package name | Bazel label |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
You’ll see some messages the first time you build, as the extension configures an opam switch, possibly creating it and/or installing missing packages; for example:
Fetching module extension @@tools_opam+//extensions:opam.bzl%opam; Building @tools_opam//extensions/config Fetching ... @@tools_opam+//extensions:opam.bzl%opam; Creating local switch for compiler 5.3.0 at /path/to/obazl_hello 54s Fetching module extension @@tools_opam+//extensions:opam.bzl%opam; Installing pkg ounit2 (1 of 12) 15s
You can use the verbosity and opam_verbosity attributes to get
more information; see Getting more info below.
|
Note
|
The initial build may take a while, especially if it needs to create and populate an opam switch. |
Toolchain strategy refers to the opam toolchain encompassing
opam, an opam switch containing an OCaml SDK (compilers,
tools, runtimes, standard library, etc.), and a set of opam packages
installed in the switch.
The opam "toolchain" is not to be confused with the OCaml toolchains
defined by rules_ocaml, which model the four basic OCaml compiler
types: ocamlopt.opt (sys>sys), ocamlc.byte (vm>vm),
ocamlopt.byte (vm>sys), and ocamlc.opt (sys>vm).
This is the default. The extension will create the entire opam
toolchain (including opam) in your XDG_DATA_HOME directory (default:
$HOME/.local/share). In that case, it
will:
-
download opam (default version 2.3.0, overridable using the
opam_versionattribute) to$XDG_DATA_HOME/obazl/opam/<opam_version>/bin/opam -
initialize an opam root at
$XDG_DATA_HOME/obazl/opam/<opam_version>/root -
create a switch, which will go in the root (e.g.
$XDG_DATA_HOME/obazl/opam/<opam_version>/root/5.3.0) -
install your opam package dependencies in that switch
Such XDG toolchains are effectively global toolchains that are
quasi-private to Obazl. They are completely separate from your system
opam configuration. They will be shared across OBazl projects that use
toolchain = "xdg".
Setting toolchain="local" tells the extension to use the local
switch it it finds one, and create it if not. If you have specified
ocaml_version then the extension will check to see if the compiler
it uses matches and print a warning if not. If the switch is missing
required packages the extension will install them.
If you do not have a local switch, the extension will create one and install your required packages.
You can use the current global switch, even if you have a local
switch, by editing MODULE.bazel and changing toolchain="local"
to toolchain="global".
If the extension finds that the version of the compiler in the current
switch does not match what you have specified in ocaml_version, it
will print a warning but proceed with the build.
If it finds that the current switch lacks any of the packages you
require, it will print an Error message and abort the build; it will
not automatically install them. You can override this by setting the
environment variable OBAZL_FORCE_INSTALL=1.
-
Ensures the requested switch is properly configured
-
If the switch already exists (local, global, or xdg), checks the version numbers and prints a warning on mismatch
-
For local and xdg toolchains:
-
Creates the switch if needed
-
-
Checks that the required packages are installed
-
for global switchs, will not install packages by default; you can force installation by setting the env variable
OBAZL_FORCE_INSTALL=1. -
for local and xdg toolchains, installs any missing packages.
-
If your switch is already properly configured (e.g. your global switch has all the packages needed), then the extension executes no updating opam commands (but may run commands like opam var prefix etc.)
Once the requested switch is copacetic, the extension "registers" one Bazel repo for each package installed in the switch, by running a repository_rule. Repo rules are only evaluated on demand; that is, their implementation functions are executed only when they are required by a build.
The implementation of the repository rule runs a configuration tool,
written in C (srcs at extensions/config and lib), that reads the META file of the opam package and then
generates the corresponding MODULE.bazel and BUILD.bazel files
that together serve to define the repo as a proxy for the opam
package. The BUILD.bazel file contains an ocaml_import rule target
that imports the compiled files etc. in the opam switch.
Finally, the configuration tool defines symlinks in the Bazel repo linking to the files in the opam switch.
For more information see below, Inspecting the Generated Repos.
The extension will run the first time you build. Bazel aggressively
caches things, so thereafter it will not need to run, unless you
change the opam.deps instruction in MODULE.bazel. That will
invalidate the cache and trigger a rerun.
The extension runs a repository_rule for each package. This only
registers the rule with Bazel; the implementation of the rule (which is what generates the BUILD.bazel files representing the opam package to Bazel) only runs on-demand.
See When is the implementation function executed? for more information.
See also Evaluation model.
When you build with tools_opam, Bazel will only use opam resources
as configured in your MODULE.bazel file. It will ignore opam-related
environment variables, current switch, etc. Furthermore, if you use
the xdg toolchain strategy, your builds will use a switch configured
with the opam installation, in $XDG_DATA_HOME, that you specified.
It follows that running opam from the command line to interact with
the switch your are using is not correct. Instead you should always
run bazel run @opam, which will ensure that your opam commands are
properly configured to use the correct opam binary, --root, and
--switch. For example:
Usage example
$ bazel run @opam -- list ... Root module : demos_obazl opam bin : /Users/<uid>/.local/share/obazl/opam/2.3.0/bin/opam OPAMROOT : /Users/<uid>/.local/share/obazl/opam/2.3.0/root OPAMSWITCH : 5.1.1 # Packages matching: installed # Name # Installed # Synopsis alcotest 1.8.0 Alcotest is a lightweight and colourful test framework astring 0.8.5 Alternative String module for OCaml ...
The transient messages you may see as the build proceeds are logged by
Bazel. Show the location of the log file by running bazel info command_log.
An easy way to inspect the log is to define an alias before running the build:
alias "bl=less -R `bazel info command_log`"
Then $ bl will show the log. As a convenience, you can just
$ source tools/source.me
You can also ask the tools_opam extension to run more verbosely by
setting the verbosity attribute in MODULE.bazel to a value greater
than 0. For this to take effect, run $ bazel clean first.
When toolchain is set to local or xdg, the extension will
execute opam commands as needed to install and/or configure the
switch. You can inspect these commands by setting opam_verbosity to
a number greater than zero in MODULE.bazel. Setting 1 will just
print the commands; values greater than 1 will pass -vv.. to the
opam commands, where the number of v`s is `opam_verbosity - 1. For
example, setting opam_verbosity = 3 will pass -vv.
Bazel places the generated repos in the external subdirectory of the
output_base, which you can find by running $ bazel info output_base.
$ ls `bazel info output_base`/external
The repositories generated by the tools_opam extension look like this:
tools_opam+ tools_opam++opam+opam tools_opam++opam+opam.ocamlsdk tools_opam++opam+opam.ounit2 tools_opam++opam+opam.seq tools_opam++opam+opam.stdlib-shims tools_opam++opam+opam.stublibs
Note the structure: concatenation of rootmodule+,
+extension+, and repo.
|
Important
|
This is the form of "canonical" names. In this example, the
apparent name of the ounit2 repo is opam.ounit2; its canonical
name is tools_opam++opam+opam.ounit2. In a Bazel label, the former
corresponds to @opam.ounit2 (one @) and the latter is
@@tools_opam++opam+opam.ounit2 (two @@). For more information
see Repository names and strict deps and Repository names and visibility.
|
The extension derives the repo name by prefixing opam. to the opam
package name. If you prefer not to use the prefix in your build code
(e.g. you want @ounit2 rather than @opam.ounit2), you can write
(in MODULE.bazel) use_repo(opam, ounit2="opam.ounit2") instead
of use_repo(opam, "opam.ounit2"). This aliasing is local; the name
of the repo remains tools_opam++opam+opam.ounit2.
To view the symlinks created by the repo rule for ounit2:
ls `bazel info output_base`/external/tools_opam++opam+opam.ounit2/lib
You can inspect everything in the repo using standard shell tools. Alternatively, you can use Bazel’s query functionality.
bazel query @opam.ounit2//lib:all --output=build
This will print the build code for all targets in the @opam.ounit2//lib package. You can also provide a specific build target, in which case Bazel will print just the fragment of the build file:
bazel query @opam.ounit2//lib:ounit2 --output=build
You can list all the files (including cmxa, cmi, cmx etc.) that are dependencies of any target:
bazel query 'kind("source file", deps(@opam.ounit2//lib))'
This will show all files in the complete dependency graph of
@opam.ounit2//lib (which is an abbreviation of
@opam.ounit2//lib:lib, which in turn is aliased to
@opam.ounit2//lib:ounit2). In this case the sources include a
dependency on package stdlib-shims:
@@tools_opam++opam+opam.stdlib-shims//lib:stdlib_shims.cma @@tools_opam++opam+opam.stdlib-shims//lib:stdlib_shims.cmxa
To limit the list to direct file dependencies, add a depth argument
(1) to the deps function:
bazel query 'kind("source file", deps(@@tools_opam++opam+opam.ounit2//lib/..., 1))
@opam.ounit2//lib:oUnit.a
@opam.ounit2//lib:oUnit.cma
@opam.ounit2//lib:oUnit.cmi
@opam.ounit2//lib:oUnit.cmt
@opam.ounit2//lib:oUnit.cmti
@opam.ounit2//lib:oUnit.cmx
@opam.ounit2//lib:oUnit.cmxa
@opam.ounit2//lib:oUnit.cmxs
@opam.ounit2//lib:oUnit.ml
@opam.ounit2//lib:oUnit.mli
@opam.ounit2//lib:oUnit2.cmi
@opam.ounit2//lib:oUnit2.cmt
@opam.ounit2//lib:oUnit2.cmti
@opam.ounit2//lib:oUnit2.cmx
@opam.ounit2//lib:oUnit2.ml
@opam.ounit2//lib:oUnit2.mli
Many other queries are possible. For example:
Show the entire dependency list:
bazel query 'deps(@opam.ounit2//lib:ounit2)'
Show direct dependencies (depth=1):
bazel query 'deps(@opam.ounit2//lib:ounit2, 1)'
Show only the deps in the deps attribute of the target:
bazel query 'labels(deps, @opam.ounit2//lib:ounit2)' @opam.ocamlsdk//lib/unix:unix @opam.ounit2//advanced/lib:lib @@tools_opam++opam+opam.seq//lib:lib