From 9e8b31923558dc8850cb51dddbbc9566a5539c99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?C=C3=A9dric=20Barreteau?= Date: Sun, 28 Dec 2025 20:51:06 -0500 Subject: [PATCH] Improve documentation about build configuration This commit rearranges existing documentation and adds more explanation, including a comprehensive overview of the various parts (platforms, modifiers and transitions), and how they are relevant for end users. In particular, a lot of information was moved from `rule_authors/` to `concepts/`, as build configuration is important for end users. A lot of information was also deduplicated across pages. --- docs/concepts/configurations.md | 372 +++++++++++++- docs/concepts/modifiers.md | 8 +- docs/concepts/transitions.md | 59 +++ .../rule_authors/configuration_transitions.md | 14 +- docs/rule_authors/configurations.md | 465 ++++++++++-------- .../rule_authors/configurations_by_example.md | 406 --------------- docs/rule_authors/string_parameter_macros.md | 21 +- docs/rule_authors/writing_toolchains.md | 14 +- 8 files changed, 719 insertions(+), 640 deletions(-) create mode 100644 docs/concepts/transitions.md delete mode 100644 docs/rule_authors/configurations_by_example.md diff --git a/docs/concepts/configurations.md b/docs/concepts/configurations.md index 1d97f3138a3af..4c849259890ba 100644 --- a/docs/concepts/configurations.md +++ b/docs/concepts/configurations.md @@ -3,14 +3,355 @@ id: configurations title: Configurations --- -For rule authors see also: [Configurations](../rule_authors/configurations.md) - -When building a target, buck always builds it in a particular "configuration." -The configuration typically includes information like the target os, target -arch, sanitizers, opt level, etc. One way to understand the effect that a -configuration has is via the `cquery` and `uquery` commands. The cquery command -will compute the appropriate configuration for a target and display a version of -that target's attributes with the configuration applied. The `uquery` command +Build configurations are how Buck models building the same target in +different ways. This can include (but is not limited to): + +- Target architecture +- Target OS +- Optimization level/build mode +- Compiler type/version +- Language version (e.g. C++ standard version) +- Sanitizers +- Passing arbitrary flags to build tools + +When building a target, Buck always builds it in a particular +configuration. Build configurations are also sometimes called +"platforms". While technically separate, those two concepts are almost +identical. + +Build configurations are [composed](../api/build/Configuration) of a set +of constraints and a set of values. + +## Configuration constraints + +Configuration constraints are enum-like constructs. Here is an example +definitions: + +```python +# //config/BUCK + +constraint( + name = "build_mode", + default = "debug", + values = [ + "debug", + "release", + ], +) +``` + +This will generate two configuration targets: +`//config:build_mode[debug]` and `//config:build_mode[release]`. + +Note that extra contraint values can be added outside of the main +`constraint` definition: + +```python +# //somewhere/else/BUCK + +constraint_value( + name = "release_no_debug_info", + constraint_setting = "//config:build_mode", +) +``` + +Now, the three configuration targets that can be used to control the +build mode would be: + +- `//config:build_mode[debug]` +- `//config:build_mode[release]` +- `//somewhere/else:release_no_debug_info` + +Constraint values can also be grouped into larger logical pieces. +Assuming that we have also defined other constraints: + +```python +config_setting( + name = "dev", + constraint_values = [ + ":build_mode[debug]", + ":compiler[clang_21]", + ":asan[enabled]", + ], +) +``` + +Note that the prelude defines some generic constraints, e.g. under +`prelude//os:` and `prelude//cpu:`, which you might want to consider +using for interoperability. + +Once defined, this constraint can used in various ways, such as: + +- Being passed on the command line to run a build in debug or release + mode. +- Being "selected on" so that building in debug vs release mode has + different effects. +- Being used for constraining compatibility (e.g. "this target can only + be built in release mode" for a benchmark). +- Being used to "transition" part of the build (e.g. "this target and + its dependencies are always built in release mode, regardless of the + dependent") + +## Configuration values + +`config_setting` can also include values taken from the buckconfig. +These can ease a migration from a legacy buckconfig setting to a build +constraint by allowing you to `select()` (more on that later) on known +buckconfig values: + +```python +config_setting( + name = "fastmode_enabled", + values = { + "build.fastmode": "true", + }, +) +``` + +This setting will be satisfied if the associated buckconfig matches, +i.e. if the user passes `build.fastmode=true` via the `-c`/`--config` +CLI flag, or if the following is set in the cell's `.buckconfig` file: + +```ini +[build] +fastmode = true +``` + +This feature only allows reading buckconfig values, not reading them. + +They are also incompatible with [configuration modifiers](./modifiers.md): +`--modifier :fastmode_enabled` does nothing. + +## Using configuration: `select()` + +Configurations can be used to change the build behavior based on which +value is currently active: + +```cpp +cxx_binary( + name = "bin", + srcs = ["main.cpp"], + compiler_flags = select({ + "//config:build_mode[debug]": ["-O0", "-g"], + "//config:build_mode[release]": ["-O3"], + }), +) +``` + +The above example is simplistic, and build mode compiler flags would +typically be set at the toolchain level, rather than per-target, but it +shows how build constraints can be used to change a build's behavior. + +`select()` can appear in almost all attributes, and it can be composed +with other collection types. For example, the following is valid: + +```python +cxx_library( + name = "lib", + exported_deps = [ + "//common:lib", + ] + select({ + "//config:os[linux]": ["//linux:lib"], + "//config:os[mac]": ["//mac:lib"], + # `DEFAULT` is a special value that is always available. + # In this case, we do not link against any extra libraries. + "DEFAULT": [], + }), +) +``` + +If only one condition matches, the `select()` resolves to that +condition. + +If multiple conditions match, then the select will be resolved to the +"most refined" of the conditions that match. A set of constraints (as in +a `config_setting`) is said to "refine" another if it is a superset of +that other's constraints. The "most refined" of a set is then the +condition that refines all the others. + +Note that `select()` is resolved during configuration. This happens +after the evaluation of the BUCK file is completed, and so Starlark code +run during BUCK file evaluation does not have access to the resolved +value. This can make it difficult to have macros that do extensive +modification or inspection of attributes (which should be done in rules +instead). However, some functions +([`select_map`](../api/build/#select_map) and +[`select_test`](../api/build/#select_test)) allow performing +limited operations on these objects. + +## Using configuration: compatibility + +Constraints can also be used to limit target compatibility. For example, +assuming that our repo supports C++20, C++23 and C++26: + +```python +# Reflection is only available starting with C++26, so we require it. +cxx_library( + name = "uses_reflection", + exported_headers = ["foo.h"], + target_compatible_with = ["//:cxx_standard[26]"] +) + +# Deducing this is not available in C++20, so we make it incompatible. +cxx_library( + name = "uses_deducing_this", + exported_headers = ["foo.h"], + target_compatible_with = select({ + "//:cxx_standard[20]": ["prelude//:none"], + "DEFAULT": [], + }) +) +``` + +Target compatibility requires all transitive dependencies to be +compatible as well. In other words, a node is compatible if and only if +the node itself and all of its transitive dependencies are compatible. +In the usual cases of a dependency via `attrs.dep()`, a target's +dependency will be configured and then checked for compatibility with +the same configuration as the dependent target. + +When trying to build a target with the wrong configuration (we will see +how shortly), the build will just fail (unless +`--skip-incompatible-targets` is passed). + +When trying to build a set of targets using a +[pattern](./target_pattern)) (e.g. `//some/package:` or +`//some/package/...`), Buck will simply ignore incompatible targets. + +See the [reference +documentation](../api/build/Select/#target_compatible_with) for more +information. + +## Changing the build configuration + +The build configuration is determined as follows: + +1. A base platform is resolved: + 1. If the user passed `--target-platforms` via the CLI, use that. + 2. Else, if the target being built has a `default_target_platform` + attribute, use that. Note that since it is used to determine the + configuration, it is one of the few attributes that are not + `select`able. + 3. Else, use the default (`parser.target_platform_detector_spec` in + the `.buckconfig` file). +2. [Configuration modifiers](./modifiers.md) are applied. Those are a + lightweight way to add ad-hoc constraints to an existing + configuration (e.g. "build with the default configuration/platform, + except with a different compiler"). +3. [Configuration transitions](./transitions.md) are applied. Those + allow changing the configuration of parts of the build graph based on + arbitrary logic (e.g. "this part of the build graph should always be + built in release mode"). + +The target platform resolution is not applied to all nodes in the graph. +Once the top-level nodes have been configured via the target platform +resolution, the configuration is propagated to dependencies (possibly +altered by transitions). + +For example: + +```sh +# Build this target with the default configuration. +buck2 build :my_target +# Build it with an entirely different configuration. +buck2 build :my_target --target-platforms //my/other:platform +# Build it with the default configuration, plus release mode. +buck2 build :my_target?release +# Equivalent to the above, but applies to all targets if multiple were built. +buck2 build :my_target -m release +``` + +See the [configurations for author](../rule_authors/configurations.md) +page for information on how to define a platform. + +Other example: + +```python +java_binary( + name = "cats", + default_target_platform = "//platforms:windows-arm64-dev", + deps = ["//libs:foo"], +) + +java_binary( + name = "dogs", + default_target_platform = "//platforms:mac-x86-dev", + deps = ["//libs:foo"], +) + +java_library( + name = "foo", + deps = [ + "//libs:common", + ] + select({ + "//constraints:x86": ["//libs:x86"], + "//constraints:mac-arm64": ["//libs:mac-arm64"], + "//constraints:windows-arm64": ["//libs:win-arm64"], + "DEFAULT": ["//libs:generic"], + }) +) +``` + +When running `buck2 build //binaries:cats //binaries:dogs`, the +`//binaries:cats` binary will be built in the `//platforms:windows-arm64-dev` +configuration and the `//binaries:dogs` binary will be built in the +`//platforms:mac-x86-dev` configuration. + +Each of those binaries depend on `//libs:foo`, but they will get +different versions of it as the binaries' configurations will each be +passed down to their dependencies. For `//binaries:cats`, its resolved +dependencies will include `//libs:win-arm64` and for `//binaries:dogs`, +it would contain `//libs:x86`. + +Note that `//libs:common` will be built twice, once for each +configuration. + +When running `buck2 build //binaries:cats //binaries:dogs --target-platforms +//platforms:mac-x86-opt`, both `//binaries:cats` and `//binaries:dogs` will +be built in the `//platforms:mac-x86-opt` configuration, use the same +dependencies, which would only be built once. + +## Configurations and output paths + +Since a target may appear within a build in multiple different configurations, +output paths cannot be derived based on just targets (as multiple actions would +map to the same outputs). For this reason, the target and the configuration are +encoded into output paths. The configuration is currently represented as a hash +of its values (a "hashed buck-out"). + +## Target platform vs execution platform + +Buck distinguishes two kinds of platforms: "regular" ones (where your +code will run), and the ones used to run compilers and other tools. +Those are distinct because it is typical to want build tools to use a +different build configuration. For example, you may want a compiler to +be built/run in release mode, even when building debug +targets. + +For this reason, Buck requires both _target_ platforms and _execution_ +platforms to be defined. The execution platforms are specified via the +`build.execution_platforms` value in `.buckconfig`. + +## Queries + +### Getting configuration constraints from its hash + +Build configurations are uniquely identified by their hash, which is not +human friendly. + +To determine what constraints are part of a configuration, run `buck2 +cquery //...` sot that Buck will discover all existing configurations, +then run `buck2 audit configurations`. + +This will list all available configurations and print their composing +contraints. + +### `cquery` and `uquery` + +One way to understand the effect that a configuration has is via the +`cquery` and `uquery` commands. The `cquery` command will compute the +appropriate configuration for a target and display a version of that +target's attributes with the configuration applied. The `uquery` command will not apply a configuration. Here is a heavily trimmed version of the outputs of invoking `uquery` and @@ -89,3 +430,18 @@ while the `nix` dependency is needed only for Linux. In `cquery` that distinction has been resolved; because the target has been configured for Linux, the `nix` dependency is present and indistinguishable from any other, while the `common-path` dependency is gone. + +## Execution groups + +Execution groups are a future feature that will allow a rule to perform +execution platform resolution multiple times and then specify in which of the +resolved platforms each action runs in. + +Traditionally, each target resolves a single execution platform. + +## See also + +- [Configuration modifiers](./modifiers.md) +- [Configuration transitions](./transitions.md) +- [Configurations for rule authors](../rule_authors/configurations.md) +- [Configuration transitions for rule authors](../rule_authors/configuration_transitions.md) diff --git a/docs/concepts/modifiers.md b/docs/concepts/modifiers.md index 726baf89b8912..630cda90ae381 100644 --- a/docs/concepts/modifiers.md +++ b/docs/concepts/modifiers.md @@ -5,8 +5,12 @@ title: Configuration modifiers Modifiers (also referred to as configuration modifiers) are a feature that lets users add [constraints](../../rule_authors/configurations) to individual -directories, target definitions and individual `buck2` invocations. They are the -recommended to customize build configuration. +directories, target definitions and individual `buck2` invocations. + +They are the recommended to customize build configurations when building +targets directly. If you need to customize parts of your build graph +(e.g. always build some specific dependencies in release mode), +[configuration transitions](transitions.md) are more appropriate. ## (Open-Source Only) Getting started with modifiers diff --git a/docs/concepts/transitions.md b/docs/concepts/transitions.md new file mode 100644 index 0000000000000..8467e8282d870 --- /dev/null +++ b/docs/concepts/transitions.md @@ -0,0 +1,59 @@ +--- +id: transitions +title: Configuration transitions +--- + +Configuration transition is a mechanism for changing the configuration when +depending on a target. + +Currently, Buck2 has incoming and outgoing transitions: + +- **Incoming** transitions are specified per-target and take effect when + depending on that target. +- **Outgoing** transitions are specified on an attribute and take effect on + dependencies that appear in that attribute. + +## Using incoming transitions + +Assuming that a transition rule called `:my_transition` exists +(see the page on [configuration transitions for rule +authors](../rule_authors/configuration_transitions.md) for how to define +one), it can be used in the following way: + +```python +cxx_library( + name = "lib1", + exported_headers = ["lib1.h"], + exported_deps = [":lib2"], + incoming_transition = [":my_transition"], +) + +cxx_library( + name = "lib2", + exported_headers = ["lib2.h"], +) +``` + +When building `lib1`, whatever the current configuration is will be +passed to `my_transition`, which will return a new configuration. This +new configuration will be used to build both `lib1` and `lib2`. + +The transition is given the current configuration and returns a new one. +For example, it could: + +- Add a new constraint to the existing configuration (e.g. add a + `release` constraint so that it and all its dependencies are built in + an optimized mode). +- Return a blank configuration with no constraints at all (useful to + deduplicate work when actions produce identical outputs, such as when + downloading artifacts). +- Make up an arbitrary configuration based on the constraints currently + set. + +## Using outgoing transitions + +Outgoing transitions are used by rule authors, not end users. + +See the page on [configuration transitions for rule +authors](../rule_authors/configuration_transitions.md) for more +information. diff --git a/docs/rule_authors/configuration_transitions.md b/docs/rule_authors/configuration_transitions.md index 302c22df6cd6e..c914fb937ec0f 100644 --- a/docs/rule_authors/configuration_transitions.md +++ b/docs/rule_authors/configuration_transitions.md @@ -3,15 +3,9 @@ id: configuration_transitions title: Configuration Transitions --- -Configuration transition is a mechanism for changing the configuration when -depending on a target. - -Currently, Buck2 has incoming and outgoing transitions: - -- **Incoming** transitions are specified per-target and take effect when - depending on that target. -- **Outgoing** transitions are specified on an attribute and take effect on - dependencies that appear in that attribute. +This page mostly focuses on how configuration transitions are +implemented. A good understanding of the high-levels +[concepts](../concepts/transitions.md) is therefore required. ## Defining transitions @@ -118,7 +112,7 @@ transition_to_watchos = rule( ) ``` -#### Idempotence +### Idempotence A transition function applied twice must produce the configuration identical to the configuration produced after applying transition once. Violating this diff --git a/docs/rule_authors/configurations.md b/docs/rule_authors/configurations.md index d81f075d718e7..b1da51c9ba0ba 100644 --- a/docs/rule_authors/configurations.md +++ b/docs/rule_authors/configurations.md @@ -4,237 +4,211 @@ title: Configurations --- This page mostly focuses on how configurations and related features are -implemented. +implemented. A good understanding of the high-level +[concepts](../concepts/configurations.md) is therefore required. -## Context +## Defining platforms -Buck configurations provide an API to express the different ways in which -projects and targets can be built. +A platform is simply a target with at least a +[`PlatformInfo`](../api/build/PlatformInfo/). That target can be an +instance of a custom rule, or it can simply use the prelude's +[`platform`](../prelude/rules/core/platform/) rule: -A configuration consists of a set of constraints and config settings (values -from buckconfig). These are determined by a base platform that sets the initial -values and then a series of transitions that may change them. - -The common way that users are exposed to configurations is in `select()` -invocations where the resolution is based on the configuration. - -A build may involve many configurations. A particular target label (`//:foo`) -may end up with multiple instances in the configured graph with different -configurations. - -## Selectable attributes - -Almost all rule attributes can be set to a `select()` value; such an attribute -is 'selectable'. These attributes' final resolved values will depend on the -configuration. - -There are some attributes that cannot use a `select()`; such attributes are -termed 'not selectable'. Examples include attributes that buck needs to read -from the unconfigured node (such as `name` and `default_target_platform`) and -attributes that are used by `platform()` rules and their dependencies (see -below). - -## Selectable resolution - -Resolving selectable attributes is pretty straightforward, it happens when -constructing the 'configured target node'. At that point, the full configuration -is available so Buck can lookup whether each constraint in the select is -satisfied or not. - -If multiple conditions of the select() match, then the select will be resolved -to the 'most refined' of the conditions that match. A set of constraints (as in -a `config_setting`) is said to 'refine' another if it is a superset of that -other's constraints. The 'most refined' of a set is then the condition that -refines all the others. If there is no 'most refined' condition of the matching -ones, it is an error. - -## Target Platform Resolution - -In the event that targets are provided on the command line, or when there is no -indication of what configuration the target will be built in, configurations are -determined by performing 'target platform resolution' on the unconfigured target -labels. - -The target platform resolution for a target `//:foo` works as follows: - -1. Look up (unconfigured) target node for `//:foo`. -1. If the command has a `--target-platforms` flag, use that. -1. If there's a `default_target_platform` attribute, use that. -1. Else, use the cell's default platform. - -This is performed independently for any targets that need a platform. Since this -resolution is done without a configuration, it means that the -`default_target_platform` attribute **is not selectable**. - -This target platform will form the initial configuration for the node. - -## Configuration propagation - -Once the top-level nodes have been configured via the target platform -resolution, the configuration is propagated to dependencies (possibly altered by -transitions). - -:::note - -The target platform resolution is not applied to all nodes in the graph. - -::: - -## Transitions - -A transition transforms a configuration by adding or changing constraint values -and config settings or by setting an entirely new underlying target platform. - -For more details, see [Configuration transitions](configuration_transitions.md). - -## `ConfigurationInfo`, `platform()` analysis, and more +```python +platform( + name = "my_platform", + constraint_values = [ + "//constraints:build_mode[debug]", + "//constraints:cpu[x64]", + ], +) +``` -The definition of a platform (either execution or target) is done with a -`platform` rule instance. The configuration is actually part of the analysis -result of the platform target (the `ConfigurationInfo` provider instance). This -is convenient from an implementation standpoint, but it leads to a situation -where some nodes are analyzed with an 'unbound' Configuration. +The configuration is actually part of the analysis result of the +platform target (the +[`ConfigurationInfo`](../api/build/ConfigurationInfo/) provider +instance). This is convenient from an implementation standpoint, but it +leads to a situation where some nodes are analyzed with an "unbound" +Configuration. All the rule types involved in defining a platform may be analyzed with an unbound configuration (`platform()`, `config_setting()`, `constraint_setting()`, -and so on). These are sometimes called 'configuration rules'. This means that +and so on). These are sometimes called "configuration rules". This means that all the attributes of these rules are not selectable. Configurations also reference a few other provider instances such as `ConstraintSettingInfo`. All of these end up being potentially produced in a context with an unbound configuration. -Using analysis for this also means that 'configuration' and 'analysis' are not +Using analysis for this also means that "configuration" and "analysis" are not distinct phases within a build (although they are still distinct for a node and are still conceptually useful). -## Configurations and output paths - -Since a target may appear within a build in multiple different configurations, -output paths cannot be derived based on just targets (as multiple actions would -map to the same outputs). For this reason, the target and the configuration are -encoded into output paths. The configuration is currently represented as a hash -of its values (a 'hashed buck-out'). - -## Target platform compatibility - -All (non-configuration) rules support a `target_compatible_with` attribute. In -addition, the rule itself can define `target_compatible_with` constraints that -affect all instances. The `target_compatible_with` attribute is a list of -constraints/config settings and it **is selectable**. - -Target platform compatibility is transitive, all _dependents_ of an incompatible -target are incompatible. In other words, a node is compatible if and only if the -node itself and all of its transitive dependencies are compatible. - -In buck, this is implemented by graph configuration returning either a -configured target node or an indicator that the node is incompatible with the -target platform. - -### Buck v1 compatibility - -Buck2 also supports the `compatible_with` field on nodes but it has different -behavior. - -In summary: - -- `compatible_with`: List of constraints, where _any_ of them must match the - configuration to be compatible. -- `target_compatible_with`: List of constraints, where _all_ of them must match - the configuration to be compatible. - -## Incompatible target skipping - -In a build-like command where a non-literal target pattern is provided (for -example, `buck build //:` or `buck build //foo/...`), the target pattern will be -resolved to a set of unconfigured targets. Those targets will then go through -[target platform resolution](#target-platform-resolution). If any of those -targets resolve to a platform where they are incompatible, building them will be -skipped. Users generally expect and prefer this behavior to needing to -explicitly specify only the targets that can build in their current context. - -If an explicitly specified literal is incompatible, it is an error. - -The implementation checks compatibility when looking up the analysis results for -configured nodes requested (in the non-ignored flow, it uses that analysis -result to lookup the default outputs and build them). - ## Execution platforms -Execution platforms/configurations are used to represent the platforms where -build execution happens. These are defined in a similar manner to target -platforms. These may or may not be what one would logically consider different -'platforms'. For example, there could be multiple different execution platforms -that all execute things similarly on the local machine. +> To Buck, both execution platforms and the list of them are based on +> `ExecutionPlatformInfo` and `ExecutionPlatformRegistrationInfo`, but +> we’ll talk in +> terms of the `execution_platform` and `execution_platforms` rules. -A build configures a fixed list of one or more execution platforms. +There are three main concepts to understand about execution platforms: -## Execution deps +1. Execution platforms +2. Execution deps +3. Execution platform resolution -Some target deps are 'execution deps'. These are the dependencies of the target -that should be built for the execution platform. For example, a compiler or -other build tool would be an execution dep. This includes all exe macro deps -(for example, `$(exe //:tool)`) and includes all `attrs.exec_dep()` deps. +### Execution platforms -## Toolchain deps +The simplest execution platform setup is the one `buck2 init` uses. This +setup gathers constraints from the host machine Buck is running on. -In addition to `attrs.exec_dep()`, there are `attrs.toolchain_dep()`, which are -similar but differ in an important way. These nodes don't select their execution -platform, but instead inherit the execution platform of whatever target -references them; hence, it must be recorded in the configured target label. In -some sense, execution platform resolution sees through them. - -In other words, `attrs.toolchain_dep()` is like a mix of `attrs.dep()` and -`attrs.exec_dep()`: - -- It inherits its target platform from the dependent build target like - `attrs.dep()` (so any `select()`s using the target of the - `attrs.toolchain_dep()` will evaluate as if they were on the target - referencing the `attrs.toolchain_dep()` - the target platform gets inherited - as with `attrs.dep()`) -- Like `attrs.exec_dep()` itself, `attrs.exec_dep()`s of the - `attrs.toolchain_dep()` target are inserted into the list of - `attrs.exec_dep()` on the dependent target of the `attrs.toolchain_dep()` - (they get passed up the dep tree, so participate in exec platform resolution). +```ini +[parser] + target_platform_detector_spec = target:root//...->prelude//platforms:default +[build] + execution_platforms = prelude//platforms:default +``` -This is illustrated in the following example: +For many projects, this will suffice, and your target and execution +platform will be the same (until you provide a config modifier, which +applies only to the target platform). But the target/exec distinction +forms the basis of all kinds of cross compilation and remote build +execution. You can use these to express "I want to compile code that +will eventually run on Windows, but all the build tools and compilers +should run on my local Linux computer". In that case the target platform +is Windows, and the execution platform is Linux. You can imagine exotic +situations in which the compiler itself has to be compiled first, on yet +another execution platform. + +More complex setups are possible. You can: + +- Add more constraints and configure code differently when it will be + executed in a build step (e.g. release mode, for faster builds of + everything else). +- Set up cross compilation, in conjunction with toolchains that will + provide the right flags. +- Let buck automatically select from multiple execution platforms + depending on what's being built (for example, most of the build can be + done on Linux, but the linker might only run on Windows). + +#### Custom execution platforms + +The process for fully specifying your own execution platforms is: + +1. Create a target that exposes an + `ExecutionPlatformRegistrationInfo(platforms = [...])` provider. Each + platform is an `ExecutionPlatformInfo`, which has its own + `ConfigurationInfo` (a set of constraints describing it, e.g. it's an + x86 server running Linux). This `ConfigurationInfo` is used for exec + platform resolution/compatibility, and also to configure software + that will run there. So often you will tell it you want all build + tools to be built themselves in release mode, so your builds are + faster. +2. Configure the `build.execution_platforms` value in your `.buckconfig` + to point to this target: + + ```ini + [build] + execution_platforms = platforms//:my_exec_platforms + ``` + +Here’s an example definition of execution platforms. ```python -target( - name = "A", - toolchain = attrs.toolchain_dep(default = ":B"), +execution_platform( + name = "mac-exec", + platform = "//platforms:mac-arm64-opt", + local_enabled = host_info().os.is_macos, + remote_enabled = True, + use_limited_hybrid = False, + remote_execution_use_case = "buck2-build", + remote_execution_properties = { + "platform": "mac-re" + }, ) -target( - name = "B", - tool = attrs.exec_dep(default = ":C") -) -``` - -The above means that `:C` will be an execution dependency of `:A` and any -`select()`s defined in `:B` would be evaluated against the same target platform -as `:A` (as target platform gets inherited by `attrs.toolchain_dep()`s). - -## Running non-execution deps - -If you have a binary that you want to run, but it isn't a build tool, then you -should use `$(exe_target //:binary)` rather than `$(exe //:binary)`. That will -run the same binary that you'd get from `buck2 build`, rather than one that is -built for the execution platform. - -The path macros vary along two axes: -- **Path Source**: either `DefaultInfo` or `RunInfo` providers -- **Configuration**: inherits the configuration or transitions to an execution - platform configuration +execution_platform( + name = "windows-exec", + platform = "//platforms:windows-arm64-opt", + local_enabled = host_info().os.is_windows, + ... +) -Specifically: +execution_platform( + name = "linux-exec", + ... +) -- `$location`: `DefaultInfo` path source, inherits configuration -- `$location_exec`: `DefaultInfo` path source, exec platform configuration -- `$exe`: `RunInfo` path source, exec platform configuration -- `$exe_target`: `RunInfo` path source, inherits configuration +execution_platforms( + name = "exec-platforms", + # In practice, may want to change this order based on the host os. + platforms = [ + "linux-exec", + "windows-exec", + "mac-exec", + ], + fallback = "error", +) +``` -## Execution platform resolution +This sets us up with three execution platforms, one for each of Windows, +macOS, and Linux. We choose a more optimized configuration for that +platform (i.e. `opt` instead of `dev`). Generally for build tools we +recommend using an optimized form, as most of the time the build will be +executing the prebuilt tools rather than building them. + +In simple cases you will only have one execution platform, and the story +ends there. + +In more complex cases, you may have multiple execution platforms. For +example, you may have a remote build farm that has both Linux and +Windows machines. When a build is requested for a particular configured +target, Buck will iterate the platforms provided in the registration +provider, and select the first platform whose configuration matches the +execution constraints. Basically, some build tools only run on Linux, so +if the tools need to be built, Buck will configure them to be built for +Linux, and then when it comes time to run them, it will schedule them to +run under the Linux execution platform. Other build tools (a +cross-platform python script) could run anywhere and these will not +influence the choice of exec platform for a given target. The +`ExecutionPlatformInfo` provider that is ultimately chosen supplies +key-value data that is sent to the remote build farm that can be used to +comply with the request, like `"OSFamily": "linux"` or a given Docker +image. You could have dozens of auto-generated execution platforms, or a +few well-known platforms that are maintained and rotated as you migrate +infrastructure over time. + +### Execution deps + +Some target deps are "execution deps". These are the dependencies of the +target that should be built for the execution platform. For example, a +compiler or other build tool would be an execution dep. This includes +all exe [string parameter macro](./string_parameter_macros.md) deps (for +example, `$(exe //:tool)`) and includes all `attrs.exec_dep()` deps. + +An exec dep differs in two ways from a normal dep: + +1. Its target platform will be set to the resolved exec platform of the + dependent. Normal deps simply inherit the target platform. +2. It influences which exec platform is chosen for a dependent target. + This is covered in exec platform resolution below. + +Aside from the way they interact with dependents, exec deps are regular +targets in the build graph. They may themselves be compiled using their +own `exec_dep`s, and therefore may need to select their own exec platform +based on their own exec deps. Each time a target somewhere in the build +graph has `exec_dep`s, Buck will do another transition through exec +platform resolution. + +You might not notice an incorrectly typed dependency edge if your only +registered execution platform = your target platform = your host machine +and you don't do much build configuration, but it matters once you start +writing your own build tools and compiling for platforms other than your +host machine. The typical error when you have misconfigured is "exec +format error" on Linux, where Buck is trying to execute e.g. a Windows +executable on a Linux machine. + +### Execution platform resolution During analysis, unlike target platform resolution, every configured node undergoes execution platform resolution independently (see exception below). @@ -288,8 +262,91 @@ def target_compatible_with(target, cfg): return True ``` -## Execution groups +## Toolchain deps + +In addition to `attrs.exec_dep()`, there is also +`attrs.toolchain_dep()`. Toolchain deps must always point to an instance +of a [toolchain rule](../concepts/toolchain.md). They are much like +[macros](custom_macros.md) for adding one or more exec deps, configuring +them a little, and storing them in a convenient provider structure. + +This intuition holds when considering configuration, as toolchains and +macros share two main properties: + +1. Toolchain deps have the same target platform as whatever uses them. + So if you `select()` in the parameters to a toolchain rule, you match + on how the dependent target was configured. Many toolchain rules + allow you to set defaults or base flags for a given target platform + in this way. +2. Toolchain deps are invisible to exec platform resolution of the + dependent target, so that the exec deps of the toolchain act as if + they are attached directly to the dependent target. They then + participate in exec platform resolution for the dependent: Buck finds + an exec platform for the dependent that is compatible with all the + tools in the toolchain. + +This has many benefits: + +- It saves you from having to put the same few `attrs.exec_dep()`s on a + bunch of different rules for the same programming language. +- It "delays" the exec_dep transition and provides a point of + configurability before this happens. +- It bundles all configurability into one spot. The user can instantiate + a toolchain at a known location (usually `toolchains//:languagename`) + that includes exec deps or paths to binaries and configures it all in + one go. + +Toolchain targets don't select their execution platform, but instead +inherit the execution platform of whatever target references them. In +some sense, execution platform resolution sees _through_ them. + +This is illustrated in the following example: + +```python +some_regular_rule( + name = "A", + toolchain = attrs.toolchain_dep(default = ":B"), +) + +some_toolchain_rule( + name = "B", + tool = attrs.exec_dep(default = ":C") +) +``` + +The above means that `:C` will be an execution dependency of `:A` and any +`select()`s defined in `:B` would be evaluated against the same target platform +as `:A` (as target platform gets inherited by `attrs.toolchain_dep()`s). + +## Visualizing configuration concepts + +### Graph with deps + +![Example graph with dependencies](/img/configurations/graph_with_deps.png) + +### Splitting //:lib3 + +As we work out the configurations here, `//:lib3` will end up being in two +different configurations, so gonna be easiest to split it now. + +### Execution Platform resolution + +![Example graph with dependencies](/img/configurations/execution_platform_resolution.png) + +This shows which nodes are involved in determining the exec configuration for +the `//:binary` target. The exec deps of `//:binary` and the exec deps for the +(transitive) toolchain deps of `//:binary` are the main things involved, that set +of exec deps must all be target compatible with an execution platform for it to +be selected. In addition, the target itself and its toolchain deps must be +`exec_compatible_with`. It is very rare to use `exec_compatible_with`, for the most +part exec platform restrictions should be marked on the tools that require the +restriction. + +### Target configurations + +![Example graph with dependencies](/img/configurations/graph_with_target_configurations.png) + +## See also -Execution groups are a future feature that will allow a rule to perform -execution platform resolution multiple times and then specify in which of the -resolved platforms each action runs in. +- [Configuration transitions for rule + authors](./configuration_transitions.md) diff --git a/docs/rule_authors/configurations_by_example.md b/docs/rule_authors/configurations_by_example.md deleted file mode 100644 index f794325776d06..0000000000000 --- a/docs/rule_authors/configurations_by_example.md +++ /dev/null @@ -1,406 +0,0 @@ ---- -id: configurations_by_example -title: Configurations By Example ---- - -[Buck’s architectural model](../../developers/architecture/buck2/) description -is a very helpful pre-read. - -The main use of configurations is changing target properties based on what the -build is targeting, which may include platform properties like OS, architecture, -runtime version (think java, python) etc and other build properties like -optimization level - -An example of how that’s done: - -```python -# //libs/BUCK - -java_library( -name = "foo", -deps = [ - "//libs:lib1", - "//libs:lib2", -] + select({ - "//constraints:x86": ["//libs:lib3-x86"], - "//constraints:mac-arm64": ["//libs:lib3-mac-arm64"], - "//constraints:windows-arm64": ["//libs:lib3-win-arm64"], - "DEFAULT": ["//libs:lib3-general"], -}) -... -) -... -``` - -- select() can appear in almost all attributes - - since example above has lists of a single element, it could’ve been a select - for a single element in the list rather than added to the list. that’s - pretty inflexible (can’t have empty cases, each case must be exactly one - element) and so it wouldn’t generally be used -- string, list, dict can all be added to select (on either side): list + select, - select + list, str + select, … -- Each branch of select() takes a config_setting (described below), which - denotes a list of required constraint_values; there’s also an optional - ”DEFAULT” branch to the select. The target platform resolution rules (below) - pick a platform, which itself gives a list of provided constraint_values. A - branch matches if all its required constraint_values are provided by the - platform. If no branch matches then the DEFAULT branch is used (or failure if - there’s no DEFAULT branch); if one branch matches it is used, if more than one - branch matches then see the “select resolution ambiguity (refinement)” section - below. -- select() is resolved during configuration. this happens after the evaluation - of the BUCK file is completed, and so starlark code run during BUCK file - evaluation does not have access to the resolved value. This can make it - difficult to have macros that do extensive modification or inspection of - attributes (and certainly we encourage doing that in rules instead). There are - some functions to do some limited operations on these objects: - - select_map(obj, function): applies function to all possible resolved values - in obj - - ex: - `select_map([1] + select({x: 2, y: 3}), lambda v: v+1) == [2] + select(x: 3, y: 4)` - - select_test(obj, function): function should return a bool, then applies - function to each resolved value and returns True if function returns True - for any of them - -## Defining Configurations - -First, define constraints and config settings. Defining constraints is done with -constraint_setting and constraint_value. constraint_setting in some sense is the -ID of a group of constraints each defined with constraint_value. In any -configuration, only one value can be present for a constraint_setting. The -config_setting rule allows creating a logical AND of constraints, and also can -require that buckconfig keys have certain values. - -```python -# //constraints/BUCK - -# constraint_setting defines a key for a logical group of constraint values. A configuration can only -# have at most one constraint value set for each constraint_settings -constraint_setting( - name = "arch", -) - -constraint_value( - name = "x86", - constraint_setting = ":arch", -) - -constraint_value( - name = "arm64", - constraint_setting = ":arch", -) - -constraint_setting( - name = "os", -) - -constraint_value( - name = "windows", - constraint_setting = ":os", -) - -constraint_value( - name = "mac", - constraint_setting = ":os", -) - -constraint_setting( - name = "mode", -) - -constraint_value( - name = "dev", - constraint_setting = ":mode", -) - -constraint_value( - name = "opt", - constraint_setting = ":mode", -) - -# can use config_setting to group constraint values into larger logical pieces -config_setting( - name = "mac-arm64", - constraint_values = [ - ":mac", - ":arm64", - ] -) - -config_setting( - name = "windows-arm64", - constraint_values = [ - ":windows", - ":arm64", - ] -) - -# an example of checking a buckconfig value. If the buckconfig is set, -# this config_setting is satisfied in all configurations -config_setting( - name = "check_some_config", - values = { - "foo.fastmode_enabled": "true", - } -) -``` - -Next, define platforms (which, confusingly, create what we call a -configuration). platforms are just a collection of constraints. A platform() can -have other platforms as deps and will union the constraints associated with that -platform. this example shows a couple techniques that can be helpful for -defining platforms - -```python -#//platforms/BUCK - -[ - platform( - name = "{}-{}".format(base, mode) - deps = [":{}".format(base)], - constraint_values = ["//constraints:{}".format(mode)] - ) - for base in ["mac-x86", "mac-arm64", "windows-x86", "windows-arm64"] - for mode in ["dev", "opt"] -] - -[ - platform( - name = name, - constraint_values = constraint_values - ) for name, constraint_values in [ - ("mac-x86", ["//constraints:mac", "//constraints:x86"]), - ("mac-arm64", ["//constraints:mac", "//constraints:arm64"]), - ("windows-x86", ["//constraints:windows", "//constraints:x86"]), - ("windows-arm64", ["//constraints:windows", "//constraints:arm64"]), - ] -] -``` - -## Target Platform Resolution - -The one remaining piece to put these all together is about selecting a target -platform for the top-level targets. - -In the case that targets are provided on the command line, configurations are -determined by performing 'target platform resolution' on the unconfigured target -labels. - -The target platform resolution for a target //:foo works as follows: - -1. Look up (unconfigured) target node for //:foo. -1. If the command has a --target-platforms flag, use that. -1. If there's a default_target_platform attribute on the node, use that. -1. Else, use the cell's default platform spec (from buckconfig - parser.target_platform_detector_spec). - -This is performed independently for any top-level targets that need a platform. -Since this resolution is done without a configuration, it means that the -default_target_platform attribute is not selectable. - -This target platform will form the initial configuration for the node and will -be passed down to all of the target dependencies of that node (exceptions, like -exec deps, are described below). - -Example: - -```python -# //binaries/BUCK - -java_binary( - name = "cats", - default_target_platform = "//platforms:windows-arm64-dev", - deps = ["//libs:foo"], -) - -java_binary( - name = "dogs", - default_target_platform = "//platforms:mac-x86-dev", - deps = ["//libs:foo"], -) -``` - -If you then do `buck2 build //binaries:cats //binaries:dogs`, the -//binaries:cats binary will be built in the //platforms:windows-arm64-dev -configuration and the //binaries:dogs binary will be built in the -//platforms:mac-x86-dev configuration. Each of those binaries depend on -//libs:foo, but they will get different versions of it as the binaries’ -configurations will each be passed down to their dependencies. - -If you look at the //libs:foo defined above, for //binaries:cats its resolved -dependencies will include //libs:lib3-win-arm64 and for //binaries:dogs it would -contain //libs:lib3-x86. - -You can specify a different target platform on the command line. If you run - -`buck2 build //binaries:cats //binaries:dogs --target-platforms //platforms:mac-x86-opt`, -both //binaries:cats and //binaries:dogs will be built in the -//platforms:mac-x86-opt configuration. - -## Target Compatibility - -If a target doesn’t work when built targeting certain platforms or -configurations, it can specify this by setting target_compatible_with. This -attribute is a list of constraints that a configuration must have otherwise the -target will be marked as incompatible with that configuration. - -```python -# //other/BUCK - -default_target_platform = "//platforms:mac-x86-dev" if host_info().os == "mac" else "//platforms:win-x86-dev" - -... - -java_binary( - name = "other", - deps = [":other_lib"], - default_target_platform = default_target_platform, -) - -java_library( - name = "other_lib", - target_compatible_with = [ - "//constraints:dev", - "//constraints:win", - ] -) -``` - -Running `buck2 build //other:other --target-platforms //platforms:win-x86-dev` -would build other in that configuration. But running -`buck2 build //other:other --target-platforms //platforms:mac-x86-dev` would -fail, because //other:other_lib would be incompatible with that configuration -and so //other:other would be as well. buck considers it an error to request to -build (or run or install or test) an explicit target that is incompatible. - -If a package (ex //other:) or recursive (ex //other/...) pattern is provided, it -is not an error for that to include incompatible targets and they will instead -simply be skipped (buck should print a message that it is skipping them). In -this example, the default_target_platform is being selected based on the host -(you could imagine this being commonly done within some small macro layer that -your project uses). There may be other targets in the //other/BUCK file that are -compatible with mac, and so if you do `buck2 build //other:` that could build -all the targets in that package that are compatible with their -default_target_platform and if they all used the same as //other:other some of -them may be compatible with mac when building on a mac and those would be built -fine (and //other:other would be skipped). - -# Advanced topics - -## Execution Platforms - -Execution platforms are used to define the configurations and execution -properties for the platforms used by build tools during the build. Currently -there is a single list (in priority order) of all available execution platforms. -This list is provided by a target in the build.execution_platforms buckconfig -configuration key. - -> To Buck, both execution platforms and the list of them are based on -> ExecutionPlatformInfo and ExecutionPlatformRegistrationInfo, but we’ll talk in -> terms of the execution_platform and execution_platforms rules. - -There are three main concepts to understand about execution platforms: - -1. execution platforms -2. exec deps -3. execution platform resolution - -Here’s an example definition of execution platforms. - -```python -# //platforms/execution/BUCK - -execution_platform( - name = "mac-exec", - platform = "//platforms:mac-arm64-opt", - local_enabled = host_info().os.is_macos, - remote_enabled = True, - use_limited_hybrid = False, - remote_execution_use_case = "buck2-build", - remote_execution_properties = { - "platform": "mac-re" - }, -) - -execution_platform( - name = "windows-exec", - platform = "//platforms:windows-arm64-opt", - local_enabled = host_info().os.is_windows, - ... -) - -execution_platform( - name = "linux-exec", - ... -) - -execution_platforms( - name = "exec-platforms", - # in practice, may want to change this order based on the host os. - platforms = [ - "linux-exec", - "windows-exec", - "mac-exec", - ], - fallback = "error", -) -``` - -This sets us up with three execution platforms, one for each of windows, mac, -and linux. We choose a more optimized configuration for that platform (i.e. opt -instead of dev). Generally for build tools we’d recommend using an optimized -form as most of the time the build will be executing the built tools rather than -building them. - -## Exec Deps - -Exec deps are the second part of the execution platform system. An exec dep -differs in two ways from a normal dep: - -1. It will inherit the execution platform of its dependent instead of the target - platform and -1. A dependent’s execution platform will be selected so that all exec deps are - target compatible with it. - -Exec deps should be used for build tools that will be used when executing the -actions of a target. If information about the dep is going to be propagated out -of the target it almost always should not be an execution dep (except for -toolchains, see below). - -Exec deps are added primarily in two ways: - -1. By rule attributes defined with attr.exec_dep() and -1. By $(exe xxx) placeholders in attributes defined with attr.arg() - -```python -foo_rule = rule( - impl = <...> -``` - -# Visualizing Configuration Concepts - -## Graph with deps - -![Example graph with dependencies](/img/configurations/graph_with_deps.png) - -## Splitting //:lib3 - -As we work out the configurations here, //:lib3 will end up being in two -different configurations, so gonna be easiest to split it now. - -## Execution Platform resolution - -![Example graph with dependencies](/img/configurations/execution_platform_resolution.png) - -This shows which nodes are involved in determining the exec configuration for -the //:binary target. The exec deps of //:binary and the exec deps for the -(transitive) toolchain deps of //:binary are the main things involved, that set -of exec deps must all be target compatible with an execution platform for it to -be selected. In addition, the target itself and its toolchain deps must be -exec_compatible_with. It is very rare to use exec_compatible_with, for the most -part exec platform restrictions should be marked on the tools that require the -restriction. - -## Target configurations - -![Example graph with dependencies](/img/configurations/graph_with_target_configurations.png) diff --git a/docs/rule_authors/string_parameter_macros.md b/docs/rule_authors/string_parameter_macros.md index 4419769a7681a..8ab2ecdd20c73 100644 --- a/docs/rule_authors/string_parameter_macros.md +++ b/docs/rule_authors/string_parameter_macros.md @@ -37,11 +37,19 @@ filegroup( ) ``` +The target being referenced must expose a `DefaultInfo` (i.e. it must be +`buck2 build`able). + ## `$(location_exec //path/to:target)` Identical to `$(location //path/to:target)`, but the configuration is -transitioned to the execution platform. This can be useful when using `genrule` -to wrap another build system with buck. +transitioned to the execution platform (see the [page about +configuration](../concepts/configuration.md) for more information). This +can be useful when using `genrule` to wrap another build system with +buck. + +The target being referenced must expose a `DefaultInfo` (i.e. it must be +`buck2 build`able). ## `$(source relative/path/to/source)` @@ -75,10 +83,14 @@ If the `$(exe my_dependency)` dependency should actually be built with the target platform, use `$(exe_target my_dependency)` instead, which will stick to the same platform as the target. +The target being referenced must expose a `RunInfo` (i.e. it must be +`buck2 run`able). + ## `$(exe_target //path/to:target)` Identical to `$(exe //path/to:target)`, except that the target is built using -the target platform, rather than the execution platform. +the target platform, rather than the execution platform (see the [page +about configuration](../concepts/configuration.md) for more information). This is for example useful to get the paths to executables to be run as part of tests. For example: @@ -96,6 +108,9 @@ sh_test( ) ``` +The target being referenced must expose a `RunInfo` (i.e. it must be +`buck2 run`able). + ## `$(query_targets queryfunction(//path/to:target))` Runs a query on the given target and replaces the macro with the matching diff --git a/docs/rule_authors/writing_toolchains.md b/docs/rule_authors/writing_toolchains.md index e0705e7a3710f..41aa97fbf022b 100644 --- a/docs/rule_authors/writing_toolchains.md +++ b/docs/rule_authors/writing_toolchains.md @@ -28,10 +28,10 @@ are available: - Defining a custom toolchain for a language that is supported in the prelude. One can then either: - - Return the `*ToolchainInfo` provider defined in the prelude, to get - compatibility with all the prelude target rules for free. - - Return a custom toolchain provider, for use with a custom set of target - rules (which is a lot more effort). + - Return the `*ToolchainInfo` provider defined in the prelude, to get + compatibility with all the prelude target rules for free. + - Return a custom toolchain provider, for use with a custom set of target + rules (which is a lot more effort). - Defining a toolchain for a custom language/process. One then has to define a toolchain provider for it, which will be used by the rules for that language. @@ -67,7 +67,7 @@ system_cxx_toolchain( ) ``` -One would typically use [`select`](configurations_by_example.md) to customize +One would typically use [`select`](../concepts/configurations.md) to customize the toolchain e.g. based on the build mode (debug vs release) or compiler type. Note that several toolchains require you to also define a `python_bootstrap` @@ -138,5 +138,5 @@ Doing is typically as follows: [Zig-based C++ toolchain in the prelude](https://github.com/facebook/buck2/tree/main/prelude/toolchains/cxx/zig) as an example). - Expose those tools as [`RunInfo`](../../api/build/RunInfo/) providers in rules - that are referenced as [exec deps](configurations_by_example.md#exec-deps) in - a toolchain implementation. + that are referenced as [exec deps](configurations.md#execution-deps) + in a toolchain implementation.