🐛 fix(config): substitute inside override values

[gaborbernat]

An override reaches tox as a raw command line string and never went through the substitution pass every value read from a configuration file goes through. {posargs}, {env:VAR} and {env_name} therefore arrived as literal text, so tox -x 'env_run_base.commands=pytest {posargs}' -- tests src ran pytest against a directory named {posargs} rather than forwarding the arguments. This hit --override, -x and TOX_OVERRIDE, in ini and TOML alike.

An override now resolves the same substitutions a value in the configuration file would, and each format keeps its own reference syntax. set_env is the exception: it expands late for file values too, and expanding it early would change how its self-references resolve.

What an override may say is otherwise unchanged. tox still reads it with the ini string rules whatever the file format, which is what lets -x env_run_base.commands=pytest tests work in a TOML project where the file spells commands as a list of lists.

This is the part of the report that is a real defect. The { replace = "posargs", extend = true } table forwards positional arguments correctly on 4.60.1 today, and the "{posargs}" string form in a TOML list keeps resolving to one shell quoted argument, as documented.

Fixes #4047

[nathanwilliams-ct]

| and so does the { replace = "posargs", extend = true } table

That doesn't split the args either, it's only the default case that currently works when you pass a list.

Currently we have:

config (toml)	cmd	Current Result	Expected	Match
{ replace = "posargs", extend = true }	tox -- tests src	pytest 'tests src'	pytest tests src	❌
{ replace = "posargs", extend = true , default = ["tests", "banana_tests"] }	tox	pytest tests banana_tests	pytest tests banana_tests	✅
{ replace = "posargs", extend = true , default = ["tests", "banana_tests"] }	tox -- tests src	pytest 'tests src'	pytest tests src	❌
"{posargs}"	tox -- tests src	pytest 'tests src'	pytest tests src	❌
"{posargs: tests banana_tests}"	tox	pytest 'tests banana_tests'	pytest tests banana_tests	❌
"{posargs: tests banana_tests}"	tox -- tests src	pytest 'tests src'	pytest tests src	❌

config (ini)	cmd	Current Result	Expected	Match
"{posargs}"	tox -- tests src	pytest tests src	pytest tests src	✅
"{posargs: tests banana_tests}"	tox	pytest tests banana_tests	pytest tests banana_tests	✅
"{posargs: tests banana_tests}"	tox -- tests src	pytest tests src	pytest tests src	✅

[gaborbernat]

Within TOML the "{posargs: tests banana_tests}" is purposfully not supported. Use { replace = "posargs", extend = true }. The first 3 examples of you should work though.

[gaborbernat]

Rows 1 and 3 do not reproduce for me. The replace table splits the arguments on released 4.60.1:

env_list = ["a"]
[env_run_base]
skip_install = true
commands = [["pytest", { replace = "posargs", extend = true }]]

$ uvx --from 'tox==4.60.1' tox c -e a -k commands -- tests src
[testenv:a]
commands = pytest tests src

Same result with default = ["tests", "banana_tests"] added, with a literal -- in front of the table, and in pyproject.toml under [tool.tox]. If you have a config where it collapses, please paste it, since that would be a separate bug from this one. One shape that does fail is a flat commands = ["pytest", { ... }], but that errors with 'pytest' is not list rather than joining the arguments.

The three {posargs} string rows are real, and this PR fixes all of them. Against this branch:

config	cmd	result
"{posargs}"	tox -- tests src	pytest tests src
"{posargs: tests banana_tests}"	tox	pytest tests banana_tests
"{posargs: tests banana_tests}"	tox -- tests src	pytest tests src

The rule is that a list entry consisting of nothing but a posargs reference expands to one entry per argument. A reference embedded in a longer string ("--opt={posargs}") or used as a scalar value (change_dir = "{posargs:.}") still resolves to one shell quoted string, because there is nowhere in those to put a second argument. The reference documentation now spells this out, and recommends the replace table for new TOML configuration since it keeps the default as a list rather than a string tox has to split.