Improve documentation about build configuration

[cbarrete]

This commit rearranges existing documentation and adds more explanation, including a comprehensive overview of the various parts (platforms, modifiers and transitions), and how they are relevant for end users.

In particular, a lot of information was moved from rule_authors/ to concepts/, as build configuration is important for end users.

A lot of information was also deduplicated across pages.

[meta-codesync]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this in D89872976. (Because this pull request was imported automatically, there will not be any future comments.)

[8Keep]

@scottcao @Nero5023 I like this change. Thoughts?

[cormacrelf]

This is great work, very valuable docs. Wish this had been around 2 years ago. I have mostly small comments but one big one on exec platforms.

[cormacrelf]

If so, they cannot be used on the command line and have it set those buckconfigs. (I believe.) These config_settings can only be selected on / used to constrain.

[cbarrete]

That's right, although I think that the paragraph after the code block explains how to use this well enough?

I'm not sure what you're getting at, did you have an edit in mind?

[cormacrelf]

It does have a good explanation of how to use it, but it comes straight after a section saying "here's config_setting, you can use it to set all its constituent constraints on the command line".

So

config_setting can also include values taken from the buckconfig. These can ease a migration from a legacy buckconfig setting to a build constraint by allowing you to select() on known buckconfig values.

... This setting will be satisfied if ... etc

You can only query buckconfig with this feature, you cannot write values. If used as a CLI modifier -m root//:fastmode_enabled, this does nothing.

[cbarrete]

I see what you mean, I agree.
I've incorporated this in the commit, let me know if you have more comments.

[cormacrelf]

A question I've had for a while is, if I have a new CPU type that's not in the prelude, can I extend that with a constraint_value() target outside the prelude? Seems like you might be able to with the old constraint_setting/constraint_value way, but I never tried it. The new unified API looks less amenable to that. Would be useful to document how to do this if it's possible.

[cbarrete]

As far as I am aware, that's not possible.

I did consider documenting both APIs, but I don't want to confuse the readers (there's already a lot in here) and I assume that Meta does not want to encourage using the new API.

The question is valid though, and I suggest opening an issue about that.

[cormacrelf]

Turns out it works great in both cases.

genrule(
    name = "printout",
    out = "out.txt",
    cmd = "echo 'cpu:" + select({
        "config//cpu:x86_64": "x86_64",
        "root//:napoleon": "napoleon",
        "DEFAULT": "other",
    }) + " abc:" + select({
        "root//:abc[abc]": "abc",
        "root//:abc[def]": "def",
        "root//:extra": "extra",
        "DEFAULT": "other",
    }) + "' > $OUT",
)

constraint_value(
    name = "napoleon",
    constraint_setting = "prelude//cpu/constraints:cpu",
)

constraint(name = "abc", values = ["abc", "def"], default = "abc")
constraint_value(name = "extra", constraint_setting = ":abc")

buck2 build :printout --out=- -m root//:napoleon -m root//:extra
BUILD SUCCEEDED
cpu:napoleon abc:extra

[cbarrete]

Oh interesting! The inconsistency in syntax is... unfortunate, though.

Anyway, I've documented it as it's not obvious.

[cormacrelf]

I don't really think it's useful to think of them as different kinds of targets. It is very common to have a target used in both modes, to compile stuff, and also at runtime.

My take on this:

Buck requires you to specify both a target platform and an execution platform. The target platform is where your code will run. The execution platform is where the compilers and other build tools will run. Many different build tools have different names for these things ("host", etc), but those are Buck's names.

The simplest execution platform setup is the one buck2 init uses. This setup gathers constraints from the host machine Buck is running on.

[parser]
  target_platform_detector_spec = target:root//...->prelude//platforms:default
[build]
  execution_platforms = prelude//platforms:default

For many projects, this will suffice, and your target and execution platform will be the same (until you provide a config modifier, which applies only to the target platform). But the target/exec distinction forms the basis of all kinds of cross compilation and remote build execution. You can use these to express "I want to compile code that will eventually run on Windows, but all the build tools and compilers should run on my local linux computer". In that case the target platform is Windows, and the execution platform is Linux. You can imagine exotic situations in which the compiler itself has to be compiled first, on yet another execution platform.

So, more complex setups are possible. You can:

• Add more constraints and configure code differently when it will be executed in a build step (e.g. release mode, for faster builds of everything else)
• Set up cross compilation, in conjunction with toolchains that will provide the right flags
• Let buck automatically select from multiple execution platforms depending on what's being built (for example, most of the build can be done on Linux, but the linker might only run on Windows)

The process for fully specifying your own execution platforms is:

1. Create a target that exposes an ExecutionPlatformRegistrationInfo(platforms = [...]) provider. Each platform is an ExecutionPlatformInfo, which has its own ConfigurationInfo (a set of constraints describing it, e.g. it's an x86 server running linux). This ConfigurationInfo is used for exec platform resolution/compatibility, and also to configure software that will run there. So often you will tell it you want all build tools to be built themselves in release mode, so your builds are faster.

2. Configure the .buckconfig value build.execution_platforms to point to this target:

   [build]
   execution_platforms = platforms//:my_exec_platforms
   

In simple cases you will only have one execution platform, and the story ends there.

In more complex cases, you may have multiple execution platforms. For example, you may have a remote build farm that has both Linux and Windows machines. When a build is requested for a particular configured target, Buck will iterate the platforms provided in the registration provider, and select the first platform whose configuration matches the execution constraints. Basically, some build tools only run on Linux, so if the tools need to be built, Buck will configure them to be built for Linux, and then when it comes time to run them, it will schedule them to run under the Linux execution platform. Other build tools (a cross-platform python script) could run anywhere and these will not influence the choice of exec platform for a given target. The ExecutionPlatformInfo provider that is ultimately chosen supplies key-value data that is sent to the remote build farm that can be used to comply with the request, like "OSFamily": "linux" or a given Docker image. You could have dozens of auto-generated execution platforms, or a few well-known platforms that are maintained and rotated as you migrate infrastructure over time. See [TODO: relative link to https://buck2.build/docs/rule_authors/configurations_by_example/#exec-deps] for more information on exec platform resolution.

[cbarrete]

I don't really think it's useful to think of them as different kinds of targets. It is very common to have a target used in both modes, to compile stuff, and also at runtime.

You're right, thanks for catching that.

I have incorporated what you wrote, although I put most of it in the rules_author/ file, as I think that it is not relevant for most end users, and don't want to confuse them.

Thanks, and let me know what you think!

[cbarrete]

Thanks for the review!

[cbarrete]

As far as I am aware, that's not possible.

I did consider documenting both APIs, but I don't want to confuse the readers (there's already a lot in here) and I assume that Meta does not want to encourage using the new API.

The question is valid though, and I suggest opening an issue about that.

[cbarrete]

That's right, although I think that the paragraph after the code block explains how to use this well enough?

I'm not sure what you're getting at, did you have an edit in mind?

[cbarrete]

I don't really think it's useful to think of them as different kinds of targets. It is very common to have a target used in both modes, to compile stuff, and also at runtime.

You're right, thanks for catching that.

I have incorporated what you wrote, although I put most of it in the rules_author/ file, as I think that it is not relevant for most end users, and don't want to confuse them.

Thanks, and let me know what you think!

[cbarrete]

@cormacrelf I've left some threads open in case you wanted to comment further. Can you please resolve them if you have no more concerns about them?

[cbarrete]

Oh interesting! The inconsistency in syntax is... unfortunate, though.

Anyway, I've documented it as it's not obvious.

[cbarrete]

I see what you mean, I agree.
I've incorporated this in the commit, let me know if you have more comments.

[cormacrelf]

; hence, it must be recorded in the configured target label not quite clear what you're saying, but in any case that's buck's job, I don't need to think about putting stuff in target labels.

[cormacrelf]

Your statement at the end "execution platform resolution sees through them" is the most powerful intuition. You can go a step further and describe them as macros for groups of exec deps:

In addition to attrs.exec_dep(), there is also attrs.toolchain_dep(). Toolchain deps must always point to an instance of a toolchain rule. A toolchain rule is much like a macro for adding one or more exec deps altogether, configuring them a little, and storing them in a convenient provider structure. This is useful because often a single programming language has a compiler, a linker, an assembler, a linter, a static analyzer and more, all in separate executables, many of which will call the others in normal operation. A toolchain dep allows you to add all of these at once.

In other words, attrs.toolchain_dep() is like a mix of ...
The intuition that toolchains are like macros matches their two main properties:

1. Toolchain deps have the same target platform as whatever uses them. So if you select() in the parameters to a toolchain rule, you match on how the dependent target was configured. Many toolchain rules allow you to set defaults or base flags for a given target platform in this way.
2. Toolchain deps are invisible to exec platform resolution of the dependent target, so that the exec deps of the toolchain act as if they are attached directly to the dependent target. They then participate in exec platform resolution for the dependent: Buck finds an exec platform for the dependent that is compatible with all the tools in the toolchain.

This has many benefits:

• It saves you from having to put the same few attrs.exec_dep()s on a bunch of different rules for the same programming language.
• It "delays" the exec_dep transition and provides a point of configurability before this happens.
• It bundles all configurability into one spot. The user can instantiate a toolchain at a known location (usually toolchains//:languagename) that includes exec_deps or paths to binaries and configures it all in one go.

Toolchain rules are created by passing is_toolchain_rule = True to the rule() constructor.

[cormacrelf]

For # 1, maybe "Its target platform will be set to the resolved exec platform of the dependent. If it's going to run there, it needs to be built for that platform. Normal deps simply inherit the target platform."

For # 2, "It influences which exec platform is chosen for a dependent target. This is covered in exec platform resolution below."

Add after:

Aside from the way they interact with dependents, exec deps are regular targets in the build graph. They may themselves be compiled using their own exec_deps, and therefore may need to select their own exec platform based on their own exec deps. Each time a target somewhere in the build graph has exec_deps, Buck will do another transition through exec platform resolution.

You might not notice an incorrectly typed dependency edge if your only registered execution platform = your target platform = your host machine and you don't do much build configuration, but it matters once you start writing your own build tools and compiling for platforms other than your host machine. The typical error when you have misconfigured is "exec format error" on Linux, where Buck is trying to execute e.g. a Windows executable on a Linux machine.

[cormacrelf]

This is not a particularly possible example because the target rule must choose whether it's a toolchain rule or not. Make :B toolchain() instead of target().

[cormacrelf]

For some reason Github won't let me resolve any threads so anything I didn't comment on further is lgtm