Adding support for a new Stackbuild specific schema

[davidcassany]

This is adding a new schema type for stackbuild plugin on build commands. Descriptions provided by the --description flag are parsed to check if they include the stackbuild attribute at root level, if not it assumes this should be validated as a regular KIWI description, however if the staclbuild="true" attribute is there the description is validated against the new schema.

The new schema is essentially the same as a regular KIWI description with the difference that any section under <image> is optional. Stackbuild schema can only be used against stashed images, which implies the stashed image already includes the KIWI description used at stash time. This new stackbuild schema is essentially used to replace or add sections to the stashed description. At build time the stackbuild description is applied on top of the stashed KIWI description to create a new full KIWI description. Then the build proceeds as a regular KIWI build (including XML validation).

This is useful to apply small modifications over an existing image without having to duplicate the whole description. Imagine adding extra packages, provide a new <type>, include extra users, etc. All other files of the description (e.g. config.sh) are rsynced on top of the stashed description. Hence the original is preserved or replaced, new ones could also be included this way.

[davidcassany]

With this change I could something like the following.

Based on the tumbleweed description form the examples repository. I did the following steps:

# Preare the root-tree of the system
sudo kiwi-ng --debug system prepare \
        --description kiwi-descriptions/suse/x86_64/suse-tumbleweed --root root-to-stash
        
# Stash the prepared system
sudo kiwi-ng system stash --root root-to-stash --container-name tw-stash

# Build the stashed image with a new stackbuild description to include additional packages and modify OEM type
sudo kiwi-ng --type oem system stackbuild --stash tw-stash --target-dir new-tw-build --description demo

Where the contents of the demo folder is just a stackbuild XML description:

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="0.1" stackbuild="true" name="tumbleweed">
    <description type="system">
        <author>Marcus Schaefer</author>
        <contact>[email protected]</contact>
        <specification>
            Tumbleweed Appliance, is a small text based image including podman container runtime
        </specification>
    </description>
    <preferences>
        <type image="oem" filesystem="xfs">
            <oemconfig>
                <oem-systemsize>4096</oem-systemsize>
            </oemconfig>
        </type>
    </preferences>
    <packages type="image">
        <package name="podman"/>
    </packages>
    <packages type="oem">
        <package name="dracut-kiwi-oem-repart"/>
    </packages>
</image>

The description only includes a new podman package and dracut-kiwi-oem-repart package in case of building an oem type. In addition the oem type has been replaced by a new one requiring the kiwi-repart dracut module. Any other package type, image type still remain as they were. Hence from the given description running something like:

sudo kiwi-ng --type oem system stackbuild --stash tw-stash --target-dir new-tw-build --description demo

is still valid and results into an ISO from the stashed content including the podman package.

[davidcassany]

This PR is just to somehow finalize what I attempted to do back in a day when I started thinking around this plugin back in 2021 during a Hackweek. The xml_merge class was already coded in my local environment since then, but, at a time, I did not manage to get it fully integrated with a proper schema validation, unit testing and all that.

This might be still a rough implementation but, IMHO, good enough to be discussed, tested and eventually polished if needed without having to rework it that much.

[davidcassany]

This is the logic that I could not manage to figure out at a time, how to include the KIWI schema file during validation being sure the schema I am including is the one that will be actually in use by currently loaded module (there could be more than one KIWI installation around), so basically it is not an option to hardcode any path in the schema.rnc as I could not figure out a relative path that works for RPM installation, pip install, virtualenvs, etc.
The work around is to dynamically set the KIWI schema path at run time with the Defaults.get_schema_file method. Then use a dummy fake schema to run the convertion from rnc to rng, because trang also converts and parses any included file...

[schaefi]

@davidcassany Thanks, regarding the schema did you know that we have an <extension> section which allows you to add custom extensions to the schema ? I'm not sure if this suits you as I believe the main intention is a merge but just in case you would like to take a look here is how you could do that:

https://osinside.github.io/kiwi/contributing/schema_extensions.html

[davidcassany]

@davidcassany Thanks, regarding the schema did you know that we have an <extension> section which allows you to add custom extensions to the schema ? I'm not sure if this suits you as I believe the main intention is a merge but just in case you would like to take a look here is how you could do that:

https://osinside.github.io/kiwi/contributing/schema_extensions.html

Yes I knew about extension concept, however I do not need new addon features for what I had in mind, I needed a way to overwrite basic XML sections of an already existing image. The use case I imagined is actually having a way to build a new image based on a stacked image without the need of rewriting the full XML. Imagine I want to create a VM from a continer, then I just need to make an XML with some extra packages (kernel, systemd, etc.) and a preferences including the OEM type I want, then these sections would be included in the XML already stored in the stacked image an result into a new build including the additional packages and running the new type.
I am not fully convinced this USE case should be done the way it is implemented in this PR, probably a new subcommand would be more appropriate to distinguish builds of a different nature. I am certainly not convinced about the schema detection logic in this PR.
Take this PR as an exercise to build a new command, eventually based into a new schema without having to rewrite it from scratch, reusing all good KIWI types and elements defined for the XML schema. Also feel free to dislike it, this was for too long in my TODO list and just before Christmas I had a chance to have a look at it.

[schaefi]

Yes I knew about extension concept, however I do not need new addon features for what I had in mind, I needed a way to overwrite basic XML sections

yep makes sense, just wanted to double check but I also saw it from the code that you are basically implemented an XML merge strategy which is cool :) I'll take a closer look as soon as I get to it.

Thanks

[schaefi]

@davidcassany This is really great work 👍 👍

The only part I struggle with is that fake schema. as stackbuild has a kiwi requirement you should have kiwi in the devenv as well as in the system as a dependency later on. In kiwi I developed a method named project_file which allows to reference any file that belongs to the project. The schema file belongs to the project and can be looked up as follows

from kiwi.defaults import Defaults
print(Defaults.project_file('schema/kiwi.rnc'))

On my system with a pip installed kiwi this gives

python3.11 lala.py 
/home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc

ls -l /home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc
-rw-r--r-- 1 ms users 141916 Jan 10 14:28 /home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc

Shouldn't this work for your case too ?

[davidcassany]

In kiwi I developed a method named project_file which allows to reference any file that belongs to the project.

I saw it and this was the key to actually solve the schema validation, I am using it to dynamically replace the fake included file with the actual KIWI schema that is used at runtime. This was the key to solve the schema validation issues I had in the past.

The only reason to exist of this kiwi-fake-schema is to actually run make kiwi_stackbuild_plugin/schema.rng which in turn calls trang -I rnc -O rng <inputfile> <outputfile>. The problem is that it runs a nested syntax conversion, so it also parses and converts any included file in the input.rnc file.

So calling

make kiwi_stackbuild_plugin/schema.rng

also reads kiwi_stackbuild_plugin/kiwi-fake-schema.rnc and creates a kiwi_stackbuild_plugin/kiwi-fake-schema.rng. Using a real kiwi path here in this context is tricky, as this is just trying to convert from RNC format to RNG nothing else, I don't feel like running a dynamic path discovery here which requires active python development environment is what we want here, this is more a build/packaging thing than a runtime concern. Moreover there is the risk of overwriting and messing with conversion of the included RNC file (the actual kiwi schema from the system). I bet there might be better ways of building and managing it. Probably another option would be to simply forget about RNC and use directly the RNG XML file as the schema, since it is a simple schema I think it should be doable too.

[schaefi]

It's an option to write RNG directly. The RNC is the so called compact form of a RelaxNG schema. Most people use the compact form because it's easier to read and to maintain. But in this special case it might be ok to work with the RNG format and avoid the conversion