support setting environment variables

[mmomtchev]

This adds support for setting environment variables when running actions

Unlike the commands, which are expanded in a single LiquidJS instance, the variables are expanded in separate instances. This is probably worth discussing.

[ilg-ul]

Does npm support something similar?

[mmomtchev]

None that I can think of.

[ilg-ul]

I can't check now, but I remember when I checked some years ago, there were some references to passing configuration via shell environment variables.

I think that generally we should remain consistent.

[mmomtchev]

If you pass npm options, for example, npm --build-from-source - these will be made available to the action script in its environments as npm_config_build_from_source=true. Maybe it was this?

[mmomtchev]

The problem does not have a generic solution:
https://stackoverflow.com/questions/37140799/passing-environment-variables-in-npm-scripts

There are a number of npm packages that do this for plain npm actions.

[ilg-ul]

Unlike the commands, which are expanded in a single LiquidJS instance, the variables are expanded in separate instances. This is probably worth discussing.

Can you elaborate on the rationale behind this proposal? Do you have a use case that you can share?

[mmomtchev]

I am trying to reduce the complexity of https://github.com/mmomtchev/magickwand.js/blob/meson/package.json

Clean cross-platform method for setting the environment variables will help a lot.

I am installing custom versions of conan and meson and setting PYTHONHOME and CONAN_HOME. Your method of installing meson as a package would solve the problem for meson but there is no solution for conan.

[ilg-ul]

I am trying to reduce the complexity of ...

I see. Pretty hard to read. :-(

Can you provide a similar package.json using your solution, to see the improvement?

Clean cross-platform method for setting the environment variables will help a lot.

Aha. So the goal is to define a method for setting environment variables to be consumed by tools invoked in actions.

I have to think about it.

[ilg-ul]

BTW, as far as I understood it, the philosophy of npm is to unify all platforms by bringing links to all dependencies in the .bin folder, such that the same tool can be invoked using the same command on all platforms.

This generally works fine, except on Windows, where .cmd shims require the use of a cmd.exe sub-shell, and not all tools (for example cmake) do this.

Seeing your use case, with lots of platform specific separate definitions, made me wonder what can be done for it to benefit from a similar unifying approach? Would a mechanism for setting environment variables be enough?

[mmomtchev]

• .cmd is another problem. This comes from Node.js itself, which uses a spawn mechanism that requires the .cmd. I think that launching processes using a shell should resolve it?
• Python is, alas, unavoidable. They have made the decision to use python on Windows and python3 on macOS. Linux generally accepts both.
• Setting environment variables is very ugly since on Windows it is a separate command while on macOS / Linux it is a command prefix
• There are also some other commands such as the importing of the conan environment variables

Another solution that I was thinking of was a xPack with bash - this will also solve all cross-platform command-line problems.

[ilg-ul]

.cmd is another problem. This comes from Node.js itself, which uses a spawn mechanism that requires the .cmd. I think that launching processes using a shell should resolve it?

This solution is both a blessing and a curse. It is a blessing since it made npm on Windows possible, compensating for the lack of links to files. It is a curse since it requires the shell. For npm/xpm this is not a problem, but otherwise it is a nuisance, like having to explicitly suffix the command name with .cmd, while on other platforms the name alone is enough (I use this in all my cmake scripts to invoke the compilers, for example).

Setting environment variables is very ugly since on Windows it is a separate command while on macOS / Linux it is a command prefix

Right. And this is one aspect where your proposal makes things a bit cleaner.

... a xPack with bash - this will also solve all cross-platform command-line problems.

Yeah, possibly. There is a sh in the Windows Build Tools package, you can experiment with it, and if you find an elegant solution, we can think of a new xPack with bash. Did you experiment with a local bash (installed via msys2, for example) and it was a realistic solution?

BTW, in the npm world there is also a portable shell (shx); I use it occasionally, it seems fine.

[mmomtchev]

Alas, shx does not support setting environment variables - in fact, it has a very limited subset of the bash features.

However sh from the Windows Build Tools seems to do the job and allows using the same commands on all three major OS.

[ilg-ul]

However sh from the Windows Build Tools seems to do the job and allows using the same commands on all three major OS.

Nice! Can you share the resulting package.json that uses sh on all platforms? And possibly the one that also uses environment variables, so I have an idea where we are aiming?

If you think it is an improvement, it probably makes sense to have a bash package. It remains to be seen how difficult building it for Windows will be.

[mmomtchev]

It is only now that I understood the meson package (I discovered the meson.c file), I will try to do the same with conan.

[ilg-ul]

I will try to do the same with conan.

Well, I'm reticent to encourage you to do so. For my use case (embedded projects with extensive testing), I currently do not need any Python at all, thus a standalone meson makes some sense.

For your use case probably having a plain Python around is a more realistic solution.

Actually I do not understand the details of your build, thus a more conservative approach.

As a side note, if for my projects I'll ever need Python, I'm considering a Python3 xPack, to be installed the same on all platforms.

[mmomtchev]

@ilg-ul, I have another use case for this: my static Python xPack now comes with root CA certificates - this was a problem mainly on macOS where it required that OpenSSL 1 was installed from homebrew. Now I don't need that the root certificates are installed on macOS and Linux - Windows is not a problem since it has its standard certificate store. The problem is that I need to set the SSL_CERT_FILE environment variable. One way to do it is to use a wrapper - since I need this only on Linux and macOS, this is not that difficult: https://github.com/mmomtchev/static-portable-python/pull/11/files

However this has drawbacks and I wonder if there isn't a better way to do it. Patching Python is also an option, but this is not a trivial patch since Python will need to find the location of its own executable from within the C code of the ssl module.

[ilg-ul]

I plan to update xpm, hopefully in 2-3 weeks, and include this feature too. I just need to complete development on the documentation sites.