Ability to pass any compiler flags directly to the compiler

[connoraird]

Description

It would be very useful to pass compiler flags from the fpm.toml file directly to the compilation commands. It seems that currently it is possible to pass very specific flags such as -ffree-form but something more generic would be more flexible. For example

name = "my-fortran-project"

[fortran]
source-form = "free"
user-defined-flags = "-ffree-line-length-none"

The new user-defined-flags would then be passed straight through to the compiler and the compiler allowed to catch any errors with the flags.

Possible Solution

get_feature_flags could handle a new feature user_defined_flags

function get_feature_flags(compiler, features) result(flags)
    type(compiler_t), intent(in) :: compiler
    type(fortran_features_t), intent(in) :: features
    character(:), allocatable :: flags

    flags = ""
    if (features%implicit_typing) then
        flags = flags // compiler%get_feature_flag("implicit-typing")
    else
        flags = flags // compiler%get_feature_flag("no-implicit-typing")
    end if

    if (features%implicit_external) then
        flags = flags // compiler%get_feature_flag("implicit-external")
    else
        flags = flags // compiler%get_feature_flag("no-implicit-external")
    end if

    if (allocated(features%source_form)) then
        flags = flags // compiler%get_feature_flag(features%source_form//"-form")
    end if

    if (allocated(features%user_defined_flags)) then
        flags = flags // " " // features%user_defined_flags
    end if
end function get_feature_flags

This would of course require further changes to add user-defined-flags. Could this be added everywhere source-form also is currently?

Additional Information

No response

[perazz]

My understanding of the fortran features structure is to hide compiler details behind the flags, so I'm not 100% sure adding compiler-specific flags is the best approach here.

Regarding max line length, not all compilers support varying it but GNU definitely offers this good option.
So why don't we add it directly to the "get-feature-flags" for the free form? I think that having a generic line length is part of what a free-form source file means for most developers.

Instead of adding one more structure (remember that custom flags are passed via CLI rather than manifest), could we instead just change this:

fpm/src/fpm_compiler.F90

Line 173 in 1cfcaf8

flag_gnu_free_form = " -ffree-form", &

to this?

flag_gnu_free_form = " -ffree-form -ffree-line-length-none -ffixed-line-length-none", & 

[connoraird]

Thank you for your speedy comments!

My understanding of the fortran features structure is to hide compiler details behind the flags, so I'm not 100% sure adding compiler-specific flags is the best approach here.

This is intended to not be specific to any compiler and the responsibility then is passed to the "user". But I understand that hiding compiler specifics is a design choice and therefore shouldn't be changed lightly. However, I can still see the benefit of having a way to specify any compiler flags via the manifest, rather than requiring --flag to be passed in every command.

Regarding max line length, not all compilers support varying it but GNU definitely offers this good option. So why don't we add it directly to the "get-feature-flags" for the free form? I think that having a generic line length is part of what a free-form source file means for most developers.

I think this sounds good and would solve issues with that particular flag. I shall raise another issue/PR to update the free-form but I am still interested in this mechanism of passing any compiler flag through the manifest. Happy to be proved wrong though.

[perazz]

I agree, would probably make the most sense to have them under the [build] table? I.e. Cargo has rustflags, we may have something like this?

[build]
link = ["blas","lapack"]
flags = ["-some-other-flag"]
c-flags = ["-c-only-flag"]
cxx-flags = ["-cpp-only-flag"]

[JorgeG94]

I agree, would probably make the most sense to have them under the [build] table? I.e. Cargo has rustflags, we may have something like this?

[build]
link = ["blas","lapack"]
flags = ["-some-other-flag"]
c-flags = ["-c-only-flag"]
cxx-flags = ["-cpp-only-flag"]

I like this idea, it would make it easier to pass flags for -fopenmp and -fopenacc.

[Carltoffel]

Would it be possible to define different sets of flags for each supported compiler? I guess there are projects, that are only working with one specific compiler, or maybe don't work with one specific compiler. It would be nice to define this in the toml. This would be an opportunity for defining compiler specific flags.

E.g.

[build]
default_flags = ["-O3", "-g"]

[build.compilers.ifort]
flags = ["-O2", "-fast"]
min_version = "19.1.0"

[build.compilers.gfortran]
supported = false

[jalvesz]

One would need to cross this with the OS. Intel uses slightly different nomenclature for the same flags depending on the OS.

[perazz]

Exactly, that's why flags for shared Fortran features are standardized via the [fortran] table and internally resolved:

fpm/src/fpm_compiler.F90

Lines 160 to 174 in 1cfcaf8

	character(*), parameter :: &
	flag_gnu_coarray = " -fcoarray=single", &
	flag_gnu_backtrace = " -fbacktrace", &
	flag_gnu_opt = " -O3 -funroll-loops", &
	flag_gnu_debug = " -g", &
	flag_gnu_pic = " -fPIC", &
	flag_gnu_warn = " -Wall -Wextra", &
	flag_gnu_check = " -fcheck=bounds -fcheck=array-temps", &
	flag_gnu_limit = " -fmax-errors=1", &
	flag_gnu_external = " -Wimplicit-interface", &
	flag_gnu_openmp = " -fopenmp", &
	flag_gnu_no_implicit_typing = " -fimplicit-none", &
	flag_gnu_no_implicit_external = " -Werror=implicit-interface", &
	flag_gnu_free_form = " -ffree-form", &
	flag_gnu_fixed_form = " -ffixed-form"

so I think this structure seems preferable. For example for unlimited line length, I think it should be the default setting for the "Free" source form on all compilers (-ffree-line-length-none, -132 for Intel, etc.).

Then, one may still want compiler-based flags, I think it could make sense to have just the flag name so the initial symbol is resolved internally (e.g. because Intel wants - on unix and / on Windows):

[build]
link = ["blas","lapack"]
flags = ["some-other-flag"] # note: no dash
flags.intel = ["intel-only-flag"]
c-flags = ["c-only-flag"]
c-flags.gnu = ["gnu-only-flag"]
cxx-flags = ["cpp-only-flag"]

it would make it easier to pass flags for -fopenmp and -fopenacc.

note OpenMP is a metapackage: dependencies.openmp = "*"

I believe openacc could be implemented the same way, especially as it may carry over additional linking flags

[jalvesz]

e.g. because Intel wants - on unix and / on Windows

it would be - (linux) and /Q (windows), but there are some exception, Ex: -fpp vs /fpp for activating preprocessing with ifort, there might be others.

[lockstockandbarrel]

This seems to be a subset of allowing for profile definitions in the fpm.toml manifest file. Extensive work was being done on that at one time, and there is a related pull request but no action since that I see of. Allowing compiler options in the fpm.toml file seems like it should take into account how the package is built when it is a dependency, and which compiler and which OS as discussed;
as well as different compilations (different MPI versions, different external libraries, ...) and that project was intended to resolve those issues.
#498
Not sure why that went cold, as it was intended as a feature from the beginning; but user-defined profiles (including the default one for a package) and conditional select of which profiles seems like a good generic solution.

[emmabastas]

Would it be possible to define different sets of flags for each supported compiler? I guess there are projects, that are only working with one specific compiler, or maybe don't work with one specific compiler. It would be nice to define this in the toml. This would be an opportunity for defining compiler specific flags.

E.g.

[build]
default_flags = ["-O3", "-g"]

[build.compilers.ifort]
flags = ["-O2", "-fast"]
min_version = "19.1.0"

[build.compilers.gfortran]
supported = false

I can imagine a scenario where this contributes to fragmentation in the package ecosystem:

Let's imagine for the sake of argument that we implement the above solution. The good thing is that we now have an escape-hatch if we want to manipulate our compilers behaviors in ways not implemented by fpm. Let's also imagine (this is contrived but that's the example I came up with) that fortran.source-form = "free"was never implemented. No problem, since we have the escape-hatch a package author can simply do

[build.compilers.ifort]
flags = ["-free"]

[build.compilers.gfortran]
flags = ["-ffree-line-length-none"]

...

for all the relevant compilers. This is a little annoying, sure it would be nice if a fortran.source-form option existed to save you some time and typing, but.. we get the job done and we have more pressing issues than this little quality-of-life improvement.

Imagine down the road a package author writes a package and they use gfortran, they are kind enough to publish it for all of us to use, however, they don't bother looking up the flags equivalent to -ffree-line-length-none for other compilers to add to their manifest, maybe they forgot about it in all their haste, we end up with this manifest:

name = "A"
[build.compilers.gfortran]
flags = ["-ffree-line-length-none"]

and now if someone wants to use package A with the ifort compiler they'll run into issues.

This example is contrived, but for more complex cases, and in a scenario where many packages end up using the escape hatch I think it risks fragmenting the package ecosystem in ways that can be difficult to reverse.

[Carltoffel]

Imagine down the road a package author writes a package and they use gfortran, they are kind enough to publish it for all of us to use, however, they don't bother looking up the flags equivalent to -ffree-line-length-none for other compilers to add to their manifest, maybe they forgot about it in all their haste, we end up with this manifest:

name = "A"
[build.compilers.gfortran]
flags = ["-ffree-line-length-none"]

and now if someone wants to use package A with the ifort compiler they'll run into issues.

This example is contrived, but for more complex cases, and in a scenario where many packages end up using the escape hatch I think it risks fragmenting the package ecosystem in ways that can be difficult to reverse.

I don't think that will be a problem. Compiler profiles would probably not be used on a daily basis. And those who are not looking for it because it is necessary for them may not even know that this feature exists. If you want to use it for a project that should work everywhere, you have to take care of it accordingly. If you don't do this, there is always the option of creating a pull request or a fork.

However, compiler profiles would enable projects that would otherwise be impractical or impossible with fpm. E.g. CUDA Fortran projects.

[jalvesz]

Compiler profiles would probably not be used on a daily basis

I wouldn't say this

However, compiler profiles would enable projects that would otherwise be impractical or impossible with fpm. E.g. CUDA Fortran projects.

One does not need to go that far to see the need for profiles. Most of the time one is looking for a way of achieving the same behavior across different compilers (or close enough). Take file inlinement optimization: gfortran -flto, ifort -ipo(linux) /Qipo (windows), nvfortran -Minline. You want to compile for avx512 architecture? the flags are also different.

So, the key problem from my point of view is not achieving different behaviors for each particular compiler/platform. It is being cross-platform/compiler compatible.

[ivan-pi]

Compiler flag profiles were a GSoC 2021 project --> #498.

Jakub Jelinek maintained his development blog here: https://fortran-lang.discourse.group/t/handling-compiler-arguments-in-fpm-project-blog-by-jakub-jelinek/1297

This feature turned out to be very tricky, for several reasons:

• how to assign flags to different profiles
• what priorities do profiles take
• how to deal with different compilers
• the actions of flags across compilers might not match
• how are these flags supposed to propagate across projects
• side-effects of certain flags, e.g. -fast-math (https://simonbyrne.github.io/notes/fastmath/#flushing_subnormals_to_zero), which can affect translation units, which weren't even compiled with this flag; what this means a package adding -fast-math could break another package.
• another corner case, e.g. adding -march=native would break things for someone attempting cross-compilation for another architecture.

In CMake, you essentially end up using generator expressions based on the $<Fortran_COMPILER_ID> selector. File-specific flags are added using:

set_source_files_properties(foo.f90 PROPERTIES COMPILE_FLAGS $<...>)

If you dig, you will find several related issues,

• Managing compiler flags #112
• A Possible Solution for Compiler Flags #350

[jalvesz]

IMO:

how to assign flags to different profiles

is the key problem to solve as:

what priorities do profiles take

Once the OS and the compiler are given, fpm has no decision to take in this regard,

how to deal with different compilers
the actions of flags across compilers might not match

Should not be fpm's responsibility: the developer choses his compiler and sets the flags as per the compiler documentation. If it is wrong, then that's the developers responsibility.

side-effects of certain flags e.g. -fast-math
another corner case, e.g. adding -march=native would break things for someone attempting cross-compilation for another architecture.

I would say it is a separate issue: Should fpm build/rebuild all upstream dependencies applying the same flags? This can be debated, I see reasons to do so and reasons to don't do it. But it should be uncoupled from the possibility of defining custom profiles.

My take is, fpm should not be opinionated on any of these points, on the contrary it should enable the developer to customize his development workflow.

[ivan-pi]

Should fpm build/rebuild all upstream dependencies applying the same flags?

I guess one should have the means to force it if necessary (e.g. for debugging). This is what we have now with --flags, no?

The problem was that package A built with -ffast-math can silently change the behavior of package B that didn't use -ffast-math. There was a case of this problem in Python (https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html),

It turns out (somewhat insanely) that when -ffast-math is enabled, the compiler will link in a constructor that sets the FTZ/DAZ flags whenever the library is loaded — even on shared libraries, which means that any application that loads that library will have its floating point behavior changed for the whole process. And -Ofast, which sounds appealingly like a "make my program go fast" flag, automatically enables -ffast-math, so some projects may unwittingly turn it on without realizing the implications.

I think the issue may have been fixed in recent GCC and LLVM versions though according to discussion here (https://news.ycombinator.com/item?id=41212072).

[perazz]

Lots to think about! Thinking about it, I believe that profiles and assignment of compiler-specific flags are two different topics.

1. For compiler flags, we just need to standardize this option into fpm.toml, as now we have it but via command line and/or environment variables only. I would see it elegant to have compiler-specific parts in the [build] and [link] tables:

[build]
link = ["blas","lapack"] # all compilers
flags = ["some-other-flag"] # All compilers 
c-flags = ["c-only-flag"]
cxx-flags = ["cpp-only-flag"]
[build.intel]
flags = ["intel-only-flag"] # intel only
[build.gnu]
c-flags = ["gnu-only-flag"]
#...

1b) for build/link flags that represent Fortran-specific features, we should keep building on the facilities we have, namely metapackages (for openacc, coarrays, etc.) and the fortran table (for example, I would love to see the "free" form flag also have unlimited line length by default; we could add IEEE arithmetics-related flags to [fortran] too, etc.)

2. for profiles, the way I see it is that a "profile" would represent potentially a whole fpm.toml, with custom [build], [link], [dependencies], etc. tables. Currentely, profiles only affect the build flags but we should not limit ourselves to it. A profile is essentially a subset of commands/dependencies/macros, we should look at how cargo handles them, it is very well designed.

Here is an example I would see fit: I want to build my package with quadruple precision support (ok this may even be standardized in the Fortran table). So I imagine I will have:

[profiles]
quad-precision = [macros.cpp = "WITH_QP", build.link = "quadmath"]