Remove uninstallable revisions from lock files #929

arash77 · 2025-10-22T18:28:03Z

This could be an alternative to #838

bgruening · 2025-10-22T21:43:03Z

Thanks @arash77 can you maybe open a PR with the result of such a run.

bernt-matthias

Wonderful to have two approaches. Now we can compare. If there is disagreement we need to investigate and otherwise we can be more confident.

Maybe the output (of both approaches) should contain a reasoning why a revision is removed ...

bernt-matthias · 2025-10-23T07:17:07Z

scripts/fix_outdated.py

+            skipped += 1
+            continue
+
+        all_revs = list(uninstallable) + list(installable)


all_revs = list(revisions)?

That wouldn't include what we miss from the installable list.

Isn't uninstallable = set(revisions) - set(installable) implying revisions = uninstallable + installable.

If so, set(all_revs) = uninstallable?

What if installable has something that wasn’t in the lock file’s revisions?
Then not exactly! set(all_revs) becomes the union of uninstallable and installable, which can include extra entries not present in revisions.

For example:

revisions = ['r1', 'r2', 'r3', 'r4'] installable = ['r5', 'r2'] uninstallable = set(revisions) - set(installable) all_revs = list(uninstallable) + list(installable) print(set(all_revs)) # {'r1', 'r2', 'r3', 'r4', 'r5'} → includes 'r5' not in revisions

scripts/fix_outdated.py

bernt-matthias · 2025-10-23T07:57:13Z

scripts/fix_outdated.py

+                (
+                    cand
+                    for cand in installable
+                    if version_cache.get(cand) == cur_versions


Hrm. What if the set of tools in the repo changed? (Not sure if this is covered in my solution :))

What do you mean by changed?

If a tool was added to the tool repo, i.e. for a repo containing more than one tool.

Then should we run this regularly?

What I meant was that tools the set of tools in a tool repository might change between revisions.

Hmm, I don't know how to deal with those! Do you have any suggestions?

arash77 · 2025-10-24T10:24:33Z

Thanks @arash77 can you maybe open a PR with the result of such a run.

This is the test PR arash77#2

…able revisions

…perations

arash77 · 2025-10-24T10:36:26Z

I found out that some tools might not have any revisions to install after this run. Maybe we could add the next revision to replace them, or, if we haven’t already, handle the revisions in a separate workflow.

bernt-matthias · 2025-10-27T10:12:16Z

I found out that some tools might not have any revisions to install after this run.

How come? Do you have an example?

arash77 · 2025-10-27T10:35:05Z

How come? Do you have an example?

https://github.com/arash77/usegalaxy-eu-tools/pull/2/files#diff-e6a5a1ad82a6dcd5528e501ed88c67f70152e1587e4d73a12364afdc3b3f2825R98

bernt-matthias · 2025-10-27T10:50:31Z

The revision list of all these tools seems non-empty, i.e. there are revisions to install?

arash77 · 2025-10-27T11:04:33Z

The revision list of all these tools seems non-empty, i.e. there are revisions to install?

Yes, there are revisions to install, but not present in lock files.
The ones that have been removed are really returning errors when installed.
For example:

(134/144) Installing repository data_manager_sortmerna_database_downloader from rnateam to section "Data Managers" at revision 30bb49887172 (TRT: 0:00:28.772095)
	Repository data_manager_sortmerna_database_downloader is already installed.
	repository data_manager_sortmerna_database_downloader installed successfully (in 0:00:00.891637) at revision 30bb49887172

… and improve logging clarity

arash77 · 2025-10-28T15:15:00Z

With the new changes, it replaces the removed ones with a newer revision, even if it does not exist in the lock files.
It is also possible to replace it with scripts/update-tool.py.

bernt-matthias · 2025-10-28T16:31:18Z

I'm running my version here https://github.com/bernt-matthias/usegalaxy-eu-tools/actions/runs/18881254213/job/53884716367

since @martenson was also working on this (CESNET/galaxy_tools#21) I'm also pinging him here.

arash77 · 2025-10-29T15:21:15Z

There are some differences between your run and my run.
I could create a report for some of them:

📄 `bgruening.yaml.lock`

Tool: `flye` (owner: bgruening)

Removed revisions in Arash run:

➖ de24438c9988

Tool: `jbrowse2` (owner: fubar)

Removed revisions in Arash run:

➖ 137e08517410
➖ cec274db51c0

Added revisions in Arash run:

➕ 49f3d3878413

Tool: `flexynesis_cbioportal_import` (owner: bgruening)

Removed revisions in Arash run:

➖ 0e9e3e083350
➖ 2b414a6c35af

Added revisions in Arash run:

➕ e9a7cb5a3c63

Tool: `nanopolish_eventalign` (owner: bgruening)

Removed revisions in Arash run:

➖ 8fdb079ddaf0

📄 `cheminformatics.yaml.lock`

Tool: `gmx_energy` (owner: chemteam)

Duplicated in Matthias run:

🔁 dae135dfd519

📄 `eirene.yaml.lock`

Tool: `mfassignr_mfassign` (owner: recetox)

Duplicated in Matthias run:

🔁 4422c60a4b71

Tool: `mfassignr_mfassigncho` (owner: recetox)

Duplicated in Matthias run:

🔁 5ea9dd423109

📄 `epigenetics.yaml.lock`

Tool: `hifive` (owner: sauria)

Added revisions in Matthias run:

➕ cc95aa05f643

Added revisions in Arash run:

➕ 0a9c76d80e33

📄 `rnateam.yaml.lock`

Tool: `graphprot_predict_profile` (owner: rnateam)

Duplicated in Matthias run:

🔁 ddcf35a868b8

📄 `single-cell-ebi-gxa.yaml.lock`

Tool: `scmap_scmap_cell` (owner: ebi-gxa)

Removed revisions in Arash run:

➖ 65b719530dd5

bernt-matthias · 2025-10-29T17:03:29Z

I'm just running again. I needed to add --add in my script. Lets see :)

bernt-matthias · 2025-10-29T21:18:34Z

@arash77 we are getting closer. I looked at a few examples and in all cases I would prefer the results of my branch.

arash77 · 2025-11-07T13:10:16Z

The only problems I have with my script are these that I can’t figure out what’s going on!

`bgruening/bgruening/flye`

Removed revisions:
de24438c9988 5 -> 8d4f03b5fe9d 7 (why not 0284be52bfcf 6)

`bgruening/fubar/jbrowse2`

Removed revisions:

dc3fe98e7b37 103 -> a074cd6b5905 107 (why not 9e3f69d9fed1 104?)

`bgruening/bgruening/flexynesis_cbioportal_import`

Removed revisions:

0e9e3e083350 1 -> e9a7cb5a3c63 5 (why not ee7e61ab554d 2?)
2b414a6c35af 3 -> e9a7cb5a3c63 5 (why not 9c57d6edd3c1 4?)

`bgruening/bgruening/nanopolish_eventalign`

Removed revisions:

8fdb079ddaf0 5 -> 85a394edc247 7 (why not 855b7ba0ce9c 6?)
bee42f615f28 0 -> 0938217b3221 3 (why not c8c5caecfacc 1?)

`single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell`

Removed revisions:

65b719530dd5 12 -> 32ae4a600167 14 (why not c8c5caecfacc 13?)

bernt-matthias · 2025-11-07T14:21:14Z

bgruening/bgruening/flye
de24438c9988 5 -> 8d4f03b5fe9d 7 (why not 0284be52bfcf 6)

in the update from 6 to 7 the version was not bumped, i.e. 6 became uninstallable

bgruening/bgruening/flexynesis_cbioportal_import
0e9e3e083350 1 -> e9a7cb5a3c63 5 (why not ee7e61ab554d 2?)
2b414a6c35af 3 -> e9a7cb5a3c63 5 (why not 9c57d6edd3c1 4?)

1 = 0.2.20 + 0
2 = 0.2.20 + 0
3 = 0.2.20 + 0
4 = 0.2.20 + 1
5 = 0.2.20 + 1

so 1, 2, and 4 are uninstallable

I do not understand why 3 is removed.

Will need more time to look into this, but maybe it helps you?

bernt-matthias · 2025-11-07T15:20:51Z

bgruening/bgruening/nanopolish_eventalign

0 0.1.0
1 0.1.0
2 0.1.0 + tool-data/all_fasta.loc.sample
3 0.1.0
4 0.11.1
5 0.13.2+galaxy0
6 0.13.2+galaxy1 - tool-data/all_fasta.loc.sample
7 0.13.2+galaxy1 + tool-data/all_fasta.loc.sample

So 0->3 is expected

I do not understand why 5 is removed.

bi-gxa/scmap_scmap_cell

12 1.6.3+galaxy1
13 1.6.4+galaxy1 Invalid
14 1.6.4+galaxy2

I do not understand why 12 is removed. It is also not shown in the dropdown on the top here

arash77 · 2025-11-14T11:04:58Z

1 = 0.2.20 + 0
2 = 0.2.20 + 0
3 = 0.2.20 + 0
4 = 0.2.20 + 1
5 = 0.2.20 + 1

In the API, we don’t see tool versions in all revisions, and the tool version shown there doesn’t seem to match what you found.

revision 1 -> 0.2.20+galaxy1
revision 2 -> doesn't have any tool version
revision 3 -> 0.2.20+galaxy1
revision 4 -> doesn't have any tool version
revision 5 -> 0.2.20+galaxy1

bernt-matthias · 2025-11-18T15:58:56Z

In the API, we don’t see tool versions in all revisions, and the tool version shown there doesn’t seem to match what you found.

I basically tried to get the versions from the diffs reported in the toolshed. Probably I made some mistakes.

In the drop down menu in the TS (which I just discovered) 1 and 3 are not shown at all. If I construct the TS URL manually e.g. 3 it shows This revision can be installed: False (for 1 and 3). Wondering why this is?

The revisions 2 and 4 which can be selected show that the tools are invalid and also This revision can be installed: False.
Edit: This might also explain why you don't get versions from the API?

In summary this would explain why we go from revision 0 to 5, but I do not get yet, why 1 and 3 are not installable.

arash77 · 2025-11-18T16:18:53Z

This might also explain why you don't get versions from the API?

I think yes this is why the data is not right and it seems that it should get fixed from TS (maybe in v2) but for now it seems that it is not possible to find it out with API calls!

bgruening · 2025-11-18T20:50:12Z

I don't think we can fix the old TS and the new TS - who knows. I'm ready to be pragmatic here and if its just a few revisions that don't match implement special handling, or that I remove/hide tools

arash77 · 2025-11-19T13:45:03Z

So which way should we go? With the API or by cloning the repos?

bernt-matthias · 2025-11-23T10:21:25Z

How about using the approach implemented in this PR and storing removed revisions in an extra key of the lock files, eg uninstallable_revisions. Then we can keep track. Could also be separate files.

bgruening · 2025-11-23T12:29:38Z

Seperate file would be nice

bernt-matthias reviewed Oct 23, 2025

View reviewed changes

arash77 added 5 commits October 24, 2025 12:27

Add workflow to fix outdated tools and implement script for uninstall…

f67cb05

…able revisions

Handle errors by exiting the script in fix_outdated.py

66082d2

Remove ephemeris subproject reference

cee4a38

Implement retry logic with exponential backoff for repository fetch o…

8152656

…perations

Remove comment step from PR update workflow in fix-outdated-tools.yml

1baad47

arash77 force-pushed the fix-outdated branch from 12e062d to 1baad47 Compare October 24, 2025 10:27

Refactor tool removal logic to ensure revisions are updated correctly…

553b010

… and improve logging clarity

reverse over installable to get the latest revisions

fc78a00

Merge branch 'usegalaxy-eu:master' into fix-outdated

2de2715

Remove uninstallable revisions from lock files #929

Are you sure you want to change the base?

Remove uninstallable revisions from lock files #929

Conversation

arash77 commented Oct 22, 2025

Uh oh!

bgruening commented Oct 22, 2025

Uh oh!

bernt-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arash77 commented Oct 24, 2025

Uh oh!

arash77 commented Oct 24, 2025

Uh oh!

bernt-matthias commented Oct 27, 2025

Uh oh!

arash77 commented Oct 27, 2025

Uh oh!

bernt-matthias commented Oct 27, 2025

Uh oh!

arash77 commented Oct 27, 2025

Uh oh!

arash77 commented Oct 28, 2025

Uh oh!

bernt-matthias commented Oct 28, 2025

Uh oh!

arash77 commented Oct 29, 2025

📄 bgruening.yaml.lock

Tool: flye (owner: bgruening)

Tool: jbrowse2 (owner: fubar)

Tool: flexynesis_cbioportal_import (owner: bgruening)

Tool: nanopolish_eventalign (owner: bgruening)

📄 cheminformatics.yaml.lock

Tool: gmx_energy (owner: chemteam)

📄 eirene.yaml.lock

Tool: mfassignr_mfassign (owner: recetox)

Tool: mfassignr_mfassigncho (owner: recetox)

📄 epigenetics.yaml.lock

Tool: hifive (owner: sauria)

📄 rnateam.yaml.lock

Tool: graphprot_predict_profile (owner: rnateam)

📄 single-cell-ebi-gxa.yaml.lock

Tool: scmap_scmap_cell (owner: ebi-gxa)

Uh oh!

bernt-matthias commented Oct 29, 2025

Uh oh!

bernt-matthias commented Oct 29, 2025

Uh oh!

arash77 commented Nov 7, 2025

bgruening/bgruening/flye

bgruening/fubar/jbrowse2

bgruening/bgruening/flexynesis_cbioportal_import

bgruening/bgruening/nanopolish_eventalign

single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell

Uh oh!

bernt-matthias commented Nov 7, 2025

📄 `bgruening.yaml.lock`

Tool: `flye` (owner: bgruening)

Tool: `jbrowse2` (owner: fubar)

Tool: `flexynesis_cbioportal_import` (owner: bgruening)

Tool: `nanopolish_eventalign` (owner: bgruening)

📄 `cheminformatics.yaml.lock`

Tool: `gmx_energy` (owner: chemteam)

📄 `eirene.yaml.lock`

Tool: `mfassignr_mfassign` (owner: recetox)

Tool: `mfassignr_mfassigncho` (owner: recetox)

📄 `epigenetics.yaml.lock`

Tool: `hifive` (owner: sauria)

📄 `rnateam.yaml.lock`

Tool: `graphprot_predict_profile` (owner: rnateam)

📄 `single-cell-ebi-gxa.yaml.lock`

Tool: `scmap_scmap_cell` (owner: ebi-gxa)

`bgruening/bgruening/flye`

`bgruening/fubar/jbrowse2`

`bgruening/bgruening/flexynesis_cbioportal_import`

`bgruening/bgruening/nanopolish_eventalign`

`single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell`

bernt-matthias commented Nov 18, 2025 •

edited

Loading