Automated install tests

[ydirson]

Now based on top of preliminary PRs to separate generally-useful stuff:

• Install-driven enhancements #242
• ssh: don't use a separate logger #250
• vm cache: shorten vm description by stripping .xva suffix #251
• Install-driven enhancements, round 2 #252
• Fix VM.vdi_uuids #283

Requires at least python 3.8 and pytest 8.2.

[ydirson]

Current status: install + firstboot work in a single test
Next:

• xva export/import, split firstboot in separate test
• include answerfile in install.img
• answerfile generation

[ydirson]

Current status: 2 separate tests for install and firstboot work without any manual intervention
Next steps:

• use VM cache to gain time
• write reinstall test
• parametrize ISO remaster
• write upgrade test

[ydirson]

Current status:

• 3 separate tests for install, firstboot, "reinstall upgrade"
• test dependencies declared (requires pytest-dependency, best used with pytest-order)
• includes pytest_collection_modifyitems hook to automatically run the dependencies first

Next steps:

• write a hook to not rerun dependencies whose output is in the vm cache, starting with Add support for a hook to override skip decision. RKrahl/pytest-dependency#77
• start adding (explicit) parameters on tests
• likely postoned
  • fix pytest_collection_modifyitems hook to properly register additional tests (some fixtures are missing)
  • improve hook to deal with test parameters
  • improve hook further by not ignoring dependents of tests explicitly requested (so in a->b->c->d chain with cached VMs pytest b d will also run c

[ydirson]

Current status:

• cached VM descriptions now much less verbose, fitting in XO display
• pytest_collection_modifyitems hook for auto-registring dependencies was moved out of scope, will be a later PR
• start including some tests of the fixtures themselves
• improvements to test parametrization well under way:
  • started to make fixtures more versatile, with ability to use callables taking test parameters, leading to less duplication
  • ... but bad interaction with pytest-order, to be investigated

Next steps:

• deal with the regression on dependency ordering - may not be trivial or even doable
• finish adding uefi/bios "firmware" parameter
• add iso/net "package source" parameter
• add ability to upgrade from XS
• add ability to upgrade from 8.0-8.1 to 8.3
• add ability to upgrade from 7.5-7.6 to 8.3
• add "restore" test following "upgrade"