Testing matrix

[gforcada]

Part of #234

So far:

• split the gigantic tox.ini.j2 into more sizeable chunks
• tox environments' names are generated correctly
• tox picks the correct python version
• multiple constraints files are specified so that each environment is using the correct one
• you can override the specific testing matrix on .meta.toml (if we had not enough configuration options 😓 )
• generate the right GitHub configuration 🎉 with proper names 😆
• documentation!
• GitLab equivalent changes to the GitHub ones

[1letter]

first quick test give me an error:

$ tox -e init
init: install_deps> python -I -m pip install mxdev
init: commands[0]> mxdev -c mx.ini
###############################################################################
# Load configuration
Can not parse override: 
###############################################################################
# Read infiles
Read [r]: requirements.txt
Read [c]: constraints.txt
Read [c]: https://dist.plone.org/release/6.1-latest/constraints.txt
###############################################################################
# No sources configured!
###############################################################################
# Write outfiles
Write [c]: constraints-mxdev.txt
Write [r]: requirements-mxdev.txt
🎂 You are now ready for: pip install -r requirements-mxdev.txt
   (path to pip may vary dependent on your installation method)
init: commands[1]> echo 'Initial setup for mxdev'
init: failed with echo is not allowed, use allowlist_externals to allow it
  init: FAIL code 1 (2.57 seconds)
  evaluation failed :( (2.69 seconds)

[gforcada]

@1letter oh, sorry, it was not meant to be tested still, but indeed, there is some polishing needed. Thanks for the heads up. After the next meeting I will fix it

[stevepiercy]

Some suggestions and a question. I didn't bother to submit multiple suggestions for a single sentence or concept. Narrative docs should follow the one sentence per line guideline of Plone Documentation, and not break up concepts for semantic line breaks or PEP8 line length. Semantic line breaks are harmful, and documentation is not code.

[stevepiercy]

Suggested change

	Allow to define multiple plone and python versions in tox @gforcada
	Support combinations of multiple versions of Plone and Python in tox. @gforcada

[gforcada]

Though it is not only tox, it's being used in GHA and GitLab as well, that's why I kept it generic

[mauritsvanrees]

@gforcada I run into an error trying this.

I ran this on a branch of collective.faq where I am working on porting it to Plone 6. See collective/collective.faq#26. The current tests (without this new testing matrix support) fail while trying to run buildout on a config that does not exist. :-) So the testing matrix support could help there.

But when I run config-package on it, I get a traceback:

$ ./venv/bin/config-package --branch current /Users/maurits/zest/irceline6/src/collective.faq
*** towncrier: If you want to use Towncrier, you have to create a 'news/' folder manually.

Traceback (most recent call last):
  File "/Users/maurits/community/plone-coredev/6.2/src/meta/./venv/bin/config-package", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/maurits/community/plone-coredev/6.2/src/meta/src/plone/meta/config_package.py", line 747, in main
    package.configure()
  File "/Users/maurits/community/plone-coredev/6.2/src/meta/src/plone/meta/config_package.py", line 722, in configure
    files = method()
            ^^^^^^^^
  File "/Users/maurits/community/plone-coredev/6.2/src/meta/src/plone/meta/config_package.py", line 388, in tox
    options.update(self._handle_constraints_files(options))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/community/plone-coredev/6.2/src/meta/src/plone/meta/config_package.py", line 400, in _handle_constraints_files
    plone_versions = list(test_matrix.keys())
                          ^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'keys'

I did not yet edit .meta.toml to do anything with a test matrix yet. In fact, I have now basically emptied it from my previous manual attempt.

[mauritsvanrees]

@gforcada The problem is that if test_matrix has not been adapted in .meta.toml, the options dict will contain a default value "test_matrix" = "".

I have this change locally and then it works:

             test_matrix = options.get("test_matrix", TOX_TEST_MATRIX)
+            if not isinstance(test_matrix, dict):
+                test_matrix = TOX_TEST_MATRIX
             plone_versions = list(test_matrix.keys())

I am not sure this is the best way, but it helps. And all jobs on my PR are green, for example this one. So looking good!

[mauritsvanrees]

Ah, but the job called "6.0 on py3.13" actually has Products.CMFPlone 6.1.0. The other jobs are correct.

tox runs a pip install with -c/home/runner/work/collective.faq/collective.faq/.tox/py313-plone60/constraints.txt

Locally, .tox/py313-plone60/constraints.txt only has one line: zope.testrunner.
Ah, but locally .tox/py39-plone60/constraints.txt also has only this one line. There, just like on GitHub Actions, the py39 one is using Plone 6.0, and py313 is using 6.1.

Strange. There is some magic in tox that is not yet obvious to me.

[mauritsvanrees]

• Locally py39-plone60 has plone.staticresources 2.2.3, which should be 2.1.17.
• Locally py313-plone60 has plone.staticresources 2.2.2, which should be 2.2.0.

I don't know why those versions are higher than they should be.

[gforcada]

@stevepiercy fixes all applied, hopefully this time is all good from your side 🤞🏾

@mauritsvanrees the AttributeError should be fixed, very weird the constraints problems, let me double check as well

[gforcada]

@mauritsvanrees

Somehow the .tox/ENV/constraints.txt only has zope.testrunner and no the complete Plone 6.0/6.1 constraints file ❗

In .tox/test/constraints.txt though, the complete constraints file is there 🤔

[gforcada]

I found the problem:

The [testenv] section in tox.ini had a typo:

-deps = {[test_runner]deps}
+deps = {[base]deps}

I'm fixing it right away

[stevepiercy]

A few grammar and proper noun tweaks, and a few semantic line break habits to break, and it's good to go.

[stevepiercy]

Docs LGTM. Needs a technical review from someone more knowledgeable. Thank you!

[mauritsvanrees]

Yes, with your last fix it works on my collective.faq PR.
This is very helpful. Many thanks!