Missing sysinfo with per test config

[pevogam]

Hi all, I am using the following config option in a default avocado config file in order to enable the now available sysinfo per test:

[sysinfo.collect]
# Whether to collect sysinfo per test or just once for the entire job
per_test = True

However I find no respective sysinfo folders within the test results folder. Could it be that I am missing something obvious here?

[richtja]

Hi @pevogam, if you have sysinfo per job enabled before, then the sysinfo.collect.per_test = True should be the only thing you have to do. Can you please check if you have sysinfo.collect.enabled = True (default value) and also that the sysinfo collectibles points to the right paths on your system:

sysinfo.collectibles.commands
sysinfo.collectibles.fail_commands  
sysinfo.collectibles.fail_files
sysinfo.collectibles.files 
sysinfo.collectibles.profilers 

If all of this is correct, then you should see sysinfo collection per test in $RESULTS/test-results/$TEST/sysinfo. If you still don't get the sysinfo directory, we will have to investigate it, it might be a bug.

[pevogam]

That's great, I will take a look. However, it seems my concern would go beyond this - The complete sysinfo is still missing with our custom scheduler which brings me to two conclusions:

1. There are missing lines or something we have to do per test so that the sysinfo parameter is respected. Could you point me to something that avocado-vt or the default state machine does to set this up?

On avocado-vt side doesn't have to do anything to set up sysinfo per tests. The necessary dependencies to make this work are post test plugins' mechanism introduced in #5630 and Runnable log_dir changes introduced in #5660. Those are huge changes, but unfortunately without knowledge of your code I can't point you into a more specific place. I hope that this will help, if not we can try to investigate more deeply together.

I looked through the changes including separately at each commit in order to understand how this mechanism works internally (so far unfortunately needed to find the problem on our side) and how the sysinfo example was added. IIUC somewhere in the code new tasks are created and added as a pre- and post-test stage to collect the system info before and after the test runs. I failed to see where these tasks are created since the first set of changes does not enable sysinfo and mostly defines the plugins, entry points, and configuraiton (which should all work out of the box for an external plugin) while the second one enables sysinfo but doesn't do so in any way that would involve the state machine or internal avocado.core changes. Could you point me to a precise location where these tasks are created and I apologize if I have missed them from the changes. Also let me try to do some of the reverse pointers too - the previous limitations of the state machine (from some years ago mostly) and in particular discussion like this one meant I had to call avocado in a serial manner and have better control over the test traversal by calling each test like so:

        # TODO: use a single state machine for all test nodes when we are able
        # to at least add requested tasks to it safely (using its locks)
        await Worker(state_machine=TaskStateMachine([task], self.status_repo),
                     spawner=node.spawner, max_running=1,
                     task_timeout=job.config.get('task.timeout.running')).run()

I assume I have to create a post-task for the purpose or something like this here or find a standard mechanism to read out and queue in all pre- and post- test tasks, one test at a time. I did notice that one of your changes included something very similar to what I mentioned in the discussion above which is a great progress in making the state machine class easier to use and more flexible.

2. More importantly than the previous point, as a usual plugin developer for avocado I would expect that a custom scheduler is not supposed to be affected by anything missing and the sysinfo should be decoupled from it. This means that either on our side we are touching doing something we are not supposed to or on avocado's side there are not clear boundaries on what a plugin developer should expect to be provided with and must integrate with in return. I guess the way to figure this larger point is to look at details from (1) but ultimately I wonder if we can establish a clear boundary for what an alternative scheduler plugin could do and should not cross into.

I agree with you that there have to be clear boundaries and we are trying to do it by specific namespaces. We have avocado.core for avocado internal and avocado.plugins for plugins which has pre-defined interface which shouldn't change (might be extended, but with back port support). I know that we have problem with avocado.plugins.runner_nrunner plugin which is a runner for suites. Even that this is a plugin, it is closely connected to core and this connection might create troubles for projects which has its own SuiteRunner. And I agree we should discuss how to make boundaries here.

I am not sure what you mean by custom scheduler. If you have your own avocado.core.task.statemachine than it is part of avocado core and it is possible that avocado changes might cause trouble to you, but we do not support it. If you meant avocado.plugins.runner_nrunner plugin, then as I said that is something we have to discuss and fix. Maybe we just need to add tests which will protect avocado.plugins.runner_nrunner plugin interface.

As you can see in the example above I am generally not touching the state machine in any way and I try to stay away from internal usage (where there are also frequent changes and possibilities of unforeseen breakage) as much as possible. The way I approach the traversal is to have

from avocado.core.plugin_interfaces import SuiteRunner as RunnerInterface
...
class CartesianRunner(RunnerInterface):

which has expected runner interface (now named SuiteRunner but I needed backward compatibility) methods like run_suite and so on. More details about the runner class and the way it wraps around the state machine and inherits the runner interface can be seen here.

So if we have to advance a bit in direction of clearing such boundaries I would say that defining what to expect and what not to expect from the SuiteRunner is a good start.

[richtja]

That's great, I will take a look. However, it seems my concern would go beyond this - The complete sysinfo is still missing with our custom scheduler which brings me to two conclusions:

1. There are missing lines or something we have to do per test so that the sysinfo parameter is respected. Could you point me to something that avocado-vt or the default state machine does to set this up?

On avocado-vt side doesn't have to do anything to set up sysinfo per tests. The necessary dependencies to make this work are post test plugins' mechanism introduced in #5630 and Runnable log_dir changes introduced in #5660. Those are huge changes, but unfortunately without knowledge of your code I can't point you into a more specific place. I hope that this will help, if not we can try to investigate more deeply together.

I looked through the changes including separately at each commit in order to understand how this mechanism works internally (so far unfortunately needed to find the problem on our side) and how the sysinfo example was added. IIUC somewhere in the code new tasks are created and added as a pre- and post-test stage to collect the system info before and after the test runs. I failed to see where these tasks are created since the first set of changes does not enable sysinfo and mostly defines the plugins, entry points, and configuraiton (which should all work out of the box for an external plugin) while the second one enables sysinfo but doesn't do so in any way that would involve the state machine or internal avocado.core changes. Could you point me to a precise location where these tasks are created and I apologize if I have missed them from the changes. Also let me try to do some of the reverse pointers too - the previous limitations of the state machine (from some years ago mostly) and in particular discussion like this one meant I had to call avocado in a serial manner and have better control over the test traversal by calling each test like so:

        # TODO: use a single state machine for all test nodes when we are able
        # to at least add requested tasks to it safely (using its locks)
        await Worker(state_machine=TaskStateMachine([task], self.status_repo),
                     spawner=node.spawner, max_running=1,
                     task_timeout=job.config.get('task.timeout.running')).run()

I assume I have to create a post-task for the purpose or something like this here or find a standard mechanism to read out and queue in all pre- and post- test tasks, one test at a time. I did notice that one of your changes included something very similar to what I mentioned in the discussion above which is a great progress in making the state machine class easier to use and more flexible.

Thank you for this description. I will try to describe the pre- / post-test task mechanism more deeply. It is all based on BP004. From avocado 95.0 the RuntimeTasks are defined inside RuntimeTaskGraph which connects them base on the dependencies. The RuntimeTaskGraph is crucial for pre- / post-task mechanism, because it creates pre- / post-task and holds connections between individual tasks. You can find the creation of graph and pre- / post-test tasks in here. Basically, for each runnable, the Pre/PostRuntimeTask.get_tasks_from_test_task will call all pre- / post-test plugins to generate runnables for their pre- / post-test tasks and those runnables are used by Pre/PostRuntimeTask.from_runnable method to create runtime tasks. After this creation, all the runtime tasks are connected in dependency graph here. At the end, the dependency graph is used for sorting the tasks for state machine in nrunner plugin. I hope the explanation make sense to you, but feel free to ask me any questions to this mechanism.

2. More importantly than the previous point, as a usual plugin developer for avocado I would expect that a custom scheduler is not supposed to be affected by anything missing and the sysinfo should be decoupled from it. This means that either on our side we are touching doing something we are not supposed to or on avocado's side there are not clear boundaries on what a plugin developer should expect to be provided with and must integrate with in return. I guess the way to figure this larger point is to look at details from (1) but ultimately I wonder if we can establish a clear boundary for what an alternative scheduler plugin could do and should not cross into.

I agree with you that there have to be clear boundaries and we are trying to do it by specific namespaces. We have avocado.core for avocado internal and avocado.plugins for plugins which has pre-defined interface which shouldn't change (might be extended, but with back port support). I know that we have problem with avocado.plugins.runner_nrunner plugin which is a runner for suites. Even that this is a plugin, it is closely connected to core and this connection might create troubles for projects which has its own SuiteRunner. And I agree we should discuss how to make boundaries here.
I am not sure what you mean by custom scheduler. If you have your own avocado.core.task.statemachine than it is part of avocado core and it is possible that avocado changes might cause trouble to you, but we do not support it. If you meant avocado.plugins.runner_nrunner plugin, then as I said that is something we have to discuss and fix. Maybe we just need to add tests which will protect avocado.plugins.runner_nrunner plugin interface.

As you can see in the example above I am generally not touching the state machine in any way and I try to stay away from internal usage (where there are also frequent changes and possibilities of unforeseen breakage) as much as possible. The way I approach the traversal is to have

from avocado.core.plugin_interfaces import SuiteRunner as RunnerInterface
...
class CartesianRunner(RunnerInterface):

which has expected runner interface (now named SuiteRunner but I needed backward compatibility) methods like run_suite and so on. More details about the runner class and the way it wraps around the state machine and inherits the runner interface can be seen here.

So IIUIC you are creating every RuntimeTask here and then you are creating new TaskStateMachine for each task and you are not using the RuntimeTaskGraph for generating pre/post tasks. I think you have two options:

1. Use RuntimeTaskGraph to generate RuntimeTask for your test and all pre- / post-test tasks related to this test. I would prefer this solution because this should avoid compatibility issues in the future.
2. You can write your own solution based on the PreRuntimeTask and PostRuntimeTask classes, but then you will have to take care about compatibility issues if something will change.

So if we have to advance a bit in direction of clearing such boundaries I would say that defining what to expect and what not to expect from the SuiteRunner is a good start.

I agree.

[pevogam]

Ok this explanation was more than helpful and I have now restored sysinfo and arbitrary pre-rest and post-test tasks that might be registered via plugins.

This leaves the bigger question about the "scheduler plugin boundaries" to clear out. In particular, the approach of our suite runner is different whereby it will use its own graph and this graph while somewhat similar to the RuntimeTaskGraph above is also quite different - it considers tests as dependencies for other tests (say you would like to test a VPN configuration and its variations can be split into multiple alternative steps, each one of which is a valid test in itself), it uses test nodes and test objects that are traced in stateful composition instead of runtime tasks, and so on. What this implies for the relevant conclusions of this thread is that while I am defining a custom suite runner to inherit from the suite runner interface, this runner also has to replace the runtime graph it uses and this runtime graph does not have an interface and is part of the avocado core components (i.e. not assumed to be swappable by plugins). So I think we will have to consider the possibility of a customizable runtime graph here.

I can summarize my observations in a possible tracker issue or a clean discussion with the correct topic if you want. How does that sound?

[pevogam]

Moreover thanks to you great feature of adding tasks I was able to reuse the machine like so

@@ -152,13 +149,13 @@ class CartesianRunner(RunnerInterface):
             elif spawner == "remote":
                 # TODO: perhaps provide a brand new vanilla session for each integration test?
                 task.spawner_handle = node.get_session_to_net()
-        self.tasks += tasks
-
-        # TODO: use a single state machine for all test nodes when we are able
-        # to at least add requested tasks to it safely (using its locks)
-        await Worker(state_machine=TaskStateMachine(tasks, self.status_repo),
-                     spawner=node.spawner, max_running=1,
-                     task_timeout=job.config.get('task.timeout.running')).run()
+            await self.tsm.add_new_task(task)
+        await Worker(
+                state_machine=self.tsm,
+                spawner=node.spawner,
+                max_running=1,
+                task_timeout=job.config.get('task.timeout.running')
+            ).run()
 
     async def run_test_node(self, node):
         """

Thanks a lot for this pull request!

[pevogam]

Ok this explanation was more than helpful and I have now restored sysinfo and arbitrary pre-rest and post-test tasks that might be registered via plugins.

But it seems adding the extra pre- and post-test tasks while having all tests passing, slowed down the run by 50% or so making its duration 150% of what it used to be.

Moreover thanks to you great feature of adding tasks I was able to reuse the machine like so

This then is not even able to complete with passing tests since the asyncio seems to update the status of some 30 second long tests many minutes later, slowing everything down while also encountering other problems.

-> I guess further integration won't be as easy as I have hoped and I will have to investigate more why reusing the same state machine and creating a few supposedly quick pre- and post-test tasks deteriorates the performance so much.

[pevogam]

To answer and thus close this discussion, in order for anyone to use sysinfo and other pre- and post-test plugins with a custom scheduler plugin, they will have to construct the necessary runtime pre-tasks and post-tasks by themselves like in the following use case illustration:

         task = RuntimeTask(raw_task)
-        if spawner == "lxc":
-            task.spawner_handle = host
-        elif spawner == "remote":
-            # TODO: perhaps provide a brand new vanilla session for each integration test?
-            task.spawner_handle = node.get_session_to_net()
-        self.tasks += [task]
+        pre_tasks = PreRuntimeTask.get_tasks_from_test_task(
+            task,
+            1,
+            self.job.test_results_path,
+            None,
+            status_server_uri,
+            self.job.unique_id,
+            self.test_suite.config,
+        )
+        post_tasks = PostRuntimeTask.get_tasks_from_test_task(
+            task,
+            1,
+            self.job.test_results_path,
+            None,
+            status_server_uri,
+            self.job.unique_id,
+            self.test_suite.config,
+        )
+        tasks = [*pre_tasks, task, *post_tasks]
+        for task in tasks:
+            if spawner == "lxc":
+                task.spawner_handle = host
+            elif spawner == "remote":
+                # TODO: perhaps provide a brand new vanilla session for each integration test?
+                task.spawner_handle = node.get_session_to_net()
+        self.tasks += tasks
 
         # TODO: use a single state machine for all test nodes when we are able
         # to at least add requested tasks to it safely (using its locks)
-        await Worker(state_machine=TaskStateMachine([task], self.status_repo),
+        await Worker(state_machine=TaskStateMachine(tasks, self.status_repo),
                      spawner=node.spawner, max_running=1,
                      task_timeout=job.config.get('task.timeout.running')).run()

[pevogam]

Most of the credits here go to @richtja for explaining the relevant avocado core code.