family add and prune fix

[dwsutherland]

Related to discussion in cylc/cylc-ui#1999

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests included.
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[wxtim]

I'd really like a reproducible workflow example.

Must Have

• Duplicate test test_update_data_structure.
• test_update_data_structure failing.

Nice to have

• More comments

Take on board or ignore

(as your time and energy permit)

• Wider refactoring of test_data_store_mgr.py

[dwsutherland]

Thanks for the review Tim

I'd really like a reproducible workflow example.

Here's another example in the ticket..
cylc/cylc-ui#1999 (comment)

I'm not sure the criteria for the families being visible in the UI, but graphql will show family proxies that accumulate every cycle.. (due to how suicide triggers add and remove items)

Must Have

• Duplicate test test_update_data_structure.
• test_update_data_structure failing.

Yeah, duplicate removed.

Nice to have

• More comments

More comments added

Take on board or ignore

(as your time and energy permit)

• Wider refactoring of test_data_store_mgr.py

I've refactored the code around the tests I added.. Will leave a wider refactor to a bigger change (which there may be in the near future)

Note:
If I undo my fix, the new tests fail as expected.

[wxtim]

@dwsutherland - Unfortunately I couldn't replicate the bug with your example. 😢

[dwsutherland]

@dwsutherland - Unfortunately I couldn't replicate the bug with your example. 😢

Did you use graphiql to try view the accumulation of family proxies?

query {
  workflows(ids:["<id>"]) {
    familyProxies {
      id
    }
  }
}

with that example workflow, it was 100% reproducible ... (even though the UI doesn't show them)

[dwsutherland]

Note poll_exist has a hard coded workflow.. but should just be set the itself (strigger/run1 in this case):

[scheduling]
    initial cycle point = 20250204T2240Z
    sequential xtriggers = True
    [[xtriggers]]
        poll_exist = workflow_state(strigger/run1//%(point)s/spawn)
        poll_non_exist = workflow_state(this/workflow//%(point)s/doesnt_exist)
    [[graph]]
        PT1M = """
@wall_clock => spawn
spawn => foo & purge
@poll_non_exist => foo => fiz => fuz
foo => !purge
@poll_exist => purge
purge => !FOOS & !FIZS
"""
[runtime]
    [[root]]
        script = sleep $((1 + $RANDOM % 10))
    [[PURGEFOO]]
    [[spawn]]
        inherit = PURGEFOO
    [[FIZS]]
        inherit = PURGEFOO
    [[fiz]]
        inherit = FIZS
    [[FOOS]]
        inherit = PURGEFOO
    [[foo,fuz]]
        inherit = FOOS
    [[PURGES]]
        inherit = PURGEFOO
    [[purge]]
        inherit = PURGES

[wxtim]

@dwsutherland - Unfortunately I couldn't replicate the bug with your example. 😢

Did you use graphiql to try view the accumulation of family proxies?

query {
  workflows(ids:["<id>"]) {
    familyProxies {
      id
    }
  }
}

with that example workflow, it was 100% reproducible ... (even though the UI doesn't show them)

No. I was looking for the bug as reported. Why do the proxies accumulate but not show up in the GUI the same way.

(Yes, I saw the hardcoded inter-workflow xtrigger)

[MetRonnie]

I cannot reproduce the GraphiQL family proxies accumulating on 8.4.x (21d18ba). I tried both the query and the same thing as a subscription. I assumed strigger is the name of the workflow in your example.

Also, I'm not sure if this would be the same bug as cylc/cylc-ui#1999, which seems to be an accumulation of jobs from Tom's GraphiQL output

[oliver-sanders]

I tried to replicate using the above example and this diff:

diff --git a/flow.cylc b/flow.cylc
index 40b932b..2c80133 100644
--- a/flow.cylc
+++ b/flow.cylc
@@ -2,7 +2,7 @@
     initial cycle point = 20250204T2240Z
     sequential xtriggers = True
     [[xtriggers]]
-        poll_exist = workflow_state(strigger/run1//%(point)s/spawn)
+        poll_exist = workflow_state(%(workflow)s//%(point)s/spawn)
         poll_non_exist = workflow_state(this/workflow//%(point)s/doesnt_exist)
     [[graph]]
         PT1M = """

Unfortunately, I couldn't replicate families piling up in GraphiQL OR the empty cycles in the GUI as originally reported.

[oliver-sanders]

Have any of us managed to reproduce the "empty cycles in the GUI" issue so far?

[dwsutherland]

Have any of us managed to reproduce the "empty cycles in the GUI" issue so far?

I'm able to easily reproduce the accumulation of families, and the tests show this is fixed (if you undo the fix, but keep the tests, the tests fail)..

I'm not sure what mechanism produced a stuck UI representation, however, I'm certain that the non-pruning of families (addressed here) is to blame.. (along with some combination of w/e state/criteria the UI requires to display a node)

[dwsutherland]

Also, I'm not sure if this would be the same bug as cylc/cylc-ui#1999, which seems to be an accumulation of jobs from Tom's GraphiQL output

@MetRonnie - the main issue (in the title) is the accumulation of old cycles, which is addressed here..

The job accumulation is (I assume) a separate issue, where he may be suiciding running (or soon to be) jobs.. It's a Separate issue from cylc/cylc-ui#1999 ... He should create a new issue for job accumulation..

[MetRonnie]

I don't think the job accumulation is a separate issue to old cycles being shown in the UI. It seems the old cycles are showing as a result of the old jobs being retained in the data store. Annoyingly I have not been able to reproduce that issue either, using the tc_in_domain:noTC example he gave

[MetRonnie]

Having worked on #6656, I think this makes sense, even if none of us were able to reproduce it outside of the added integration test

[dwsutherland]

Having worked on #6656, I think this makes sense, even if none of us were able to reproduce it outside of the added integration test

Yeah, still a little surprised about this .. Maybe you didn't set n=0? I was only able doing that:
cylc/cylc-ui#1999 (comment)

Because, the corresponding task/family wouldn't be pruned in the same batch as it was added otherwise..

But thanks all the same..