Refactor Build System Internals

[mabruzzo]

Overview

This PR changes the way the internals of the build-system handles compile-time configuration parameters

Historically, every configuration choice were passed into the compiler as -D<config-var> or -D<config-var>=<val> flags. Now, we generate a header file called cholla_config.h that encodes all of the definitions, and that file is used to communicate the configuration choices to the rest of the source code.

Care has been taken so to ensure that the experience is pretty much unchanged (this is only an internal change)

I've also written a lot of website documentation, but I go into some detail below.

How it works

With the changes introduced in this PR, the build-system now

• feeds all of the -D<config-var> or -D<config-var>=<val> flags to the newly introduced configure_file.py script, which generates src/cholla_config.h from the src/cholla_config.h.in template file.
  • creation of this file is a prerequisite for compiling all object files.
  • the build-system is structured so that every call to make that doesn't invoke a PHONY target, will try to rebuild cholla_config.h. Care is taken to only overwrite the file if the configuration options have changed. (If the file does change, make will try to regenerate all of the object files)
• then, each compiler invocation that produces object files will pass -include cholla_config.h. This option makes the compiler (technically the C preprocessor) act as if the line #include "cholla_config.h" was prepended to the start of each file

Note

I'm frankly not thrilled about the use of the -include cholla_config.h flag, but after talking it over with Evan, we decided it was the best solution.
My reservations pertain to concerns over portability (while g++ and clang++ support the flag, I'm not sure about other compilers) and the fact that the inclusion is implicit.

The alternative was to actually modify ever source file in cholla (that uses a config option) and make sure that it includes the global/global.h header (which internally includes cholla_config.h) or directly includes cholla_config.h before the very first check in the file of one of the config-checks.
Issues arrive with the alternative since most conditional compilation based on whether or not a config option is defined or not.
Thus, if someone forgot to include global/global.h or cholla_config.h, the source code would just act like all of the relevant options were disabled.¹

The configure_file.py tool

This is a highly modified version of a similar script I previously contributed to grackle. This is modeled after the configure_file commands provided by CMake and Meson.

A key feature is that the program will abort if an unknown config variable is provided or if a required config variable isn't specified.

Benefits

There are a few benefits of this PR:

• the build will now fail if you make a typo in an option name
• we now have a centralized (and exhaustive) list of all build-time configuration parameters
• we are on track to stop specifying these options with extra compiler flags (we have gone from an arbitrary number of -D flags to just -include cholla_config.h, and we will hopefully be able to get rid of that in the future)
• this is also important for any future situation where we change the underlying build-system (e.g. to CMake or Meson). There are obviously no plans to make such a change, but it's conceivable that it could become necessary in the future, and if we made that change, it would be idiomatic to adopt this sort of configuration file strategy

Footnotes

1. There wouldn't be a major problem if all of the compilation options were always defined, and all conditional compilation choices were made based on the value of the option (rather than whether it is defined). In that scenario, the build would simply fail if somebody forgot an include statement. ↩

[mabruzzo]

NOTE: there are some unused configuration options in cholla_config.h.in. I clean those out in the followup to this PR: #484