canary_hpc plugin

[tjfulle]

Here's a clearer version of your pull request description while maintaining the same tone:

Overview

This pull request introduces several significant changes to the core functionality of Canary:

• It now focuses exclusively on executing test cases and not test batches.
• The resource pool is restricted to only include resources available for the current Canary invocation (i.e., resources on a compute backend are excluded).
• The batch implementation has transitioned to the canary_hpc plugin.

The second change simplifies the canary.config.resource_pool, as it no longer needs to account for resources across multiple compute nodes.

Batches

How are batches executed now? Or, how will tests be scheduled?

First, let's review the previous execution flow of canary run:

1. Identify test generators using canary_discover_generators.
2. Create test cases by locking the test generators.
3. Filter and mask test cases based on user inputs and resource availability.
4. Execute the tests using canary_runtests.

To support the third step, this PR introduces a new plugin hook:

@hookspec(firstresult=True)
def canary_resource_requirements_satisfiable(case: "TestCase") -> bool:
    """Checks if there are sufficient resources to run the specified ``case``."""
    ...

This plugin is invoked during the filtering phase to determine if the necessary resources are available for a test case. The default implementation compares the requirements of the test case with the now-simplified resource pool in Canary.

With these updates, implementing a batch runner only requires defining canary_resource_requirements_satisfiable and canary_runtests. This PR introduces the canary_hpc plugin, which accomplishes this by:

• Implementing canary_addoption to include batching arguments in the command line.
• Implementing canary_addhooks to add additional plugin hooks.
• Implementing canary_configure to inspect batch arguments and set up the hpc_connect backend (if requested).
• Implementing canary_resource_requirements_satisfiable to check if the hpc_connect backend can accommodate the test.
• Implementing canary_runtests to execute tests in batches.
• Implementing canary_runtests_startup to determine cases to run during the batch callback phase (see below).

The canary_hpc plugin processes batches similarly to how core Canary previously did. When a batch is processed, it calls back to Canary to execute the batch, meaning that running a batch is treated as a subprocess that executes canary run --bach-id=BATCH_ID. In this way, batching is managed through a hierarchical invocation of Canary.

Since core Canary no longer recognizes batches, it does not know how to handle the --batch-id argument. So, at the execution at the second level of hierarchy canary_runtests_startup handles --batch-id by reading in the batch's index and passing its cases on to canary run.

[tjfulle]

@mdmosby - this is ready to merge but I'd love to hear your thoughts on the description above.