Checkpoint/mocked workflow is not deterministic

[kinow]

Describe the bug

The generated data for the one workflow, used for the "offline" mode, and for e2e tests is not deterministic. The generated state that is snapshot'ed into the offline data may contain, for example, an expired task. But in other cases that task may not be present.

With this, every time we update the GraphQL query used (e.g. when adding deltas to gscan #543) and re-generate the offline data, we may need to update the e2e tests.

Release version(s) and/or repository branch(es) affected?

Steps to reproduce the bug

Expected behavior

Only need to update tests or code if we modify existing queries in a way that may affect the code (i.e. adding a new field most likely won't cause any issues, but removing a field or alias might).

Screenshots

Additional context

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

[kinow]

Re-generating the data this morning for #543, found that the cycle points are also not always the same.

[oliver-sanders]

Waiting for the workflow to evolve into the desired state is troublesome (why the Cylc test battery is such a pain), we should be able to do something more deterministic by borrowing the Python harness for running workflows from the Cylc Flow integration test framework.

schd = scheduler(reg)
async with run(schd):
    for state, itask in zip(TASK_STATUSES_ALL, schd.pool.get_tasks()):
        itask.state.reset(state)
    # all the tasks are in the right state now
    get_checkpoint()

Will need to wait until the event-driven deltas are in to do this:

https://github.com/cylc/cylc-flow/pull/3938/files

[kinow]

The linked PR is fixed, so I had a quick try at rewriting the UI's generate shell in Python, but many of the helper functions used in the integration tests are not exported. I started copying some of the functions into my script, but it was getting a bit more complicated than I think we'd like to maintain in Cylc UI. Probably needs some further work in Cylc Flow to export some of the utility code before we can use that here?

[hjoliver]

From @oliver-sanders

This will be even more broken once the graph view is in.
We should probably get the workflow into the desired state and shut it down.

We can then restart it from the same state at a later date (less risk of jitter).

We can then issue a GraphQL query against all of the fields the UI needs (merge all UI queries together) to get the sample data.

[oliver-sanders]

Update: With the graph view I hand-edited the sample data. TBH this is actually a pretty viable approach and, at least for the moment, easier than trying to generate this data for real. So I think ATM the best way forward is probably to hand-edit the data (see #620 (comment)) so closing this issue.

For a true end-to-end test we can investigate testing with Cypress using a real UIS and workflows started via the integration test framework. This should give a high level of determinism. This is covered by cylc/cylc-admin#33.