getObject() and getExternalInfo() load externalInfo #453

will-moore · 2025-03-13T15:30:56Z

This tweaks the base BlitzObjectWrapper._getQueryString() to include loading externalInfo.

It also adds BlitzObjectWrapper.getExternalInfo() to return the externalInfo and load it if necessary, e.g. for cases where self._obj is NOT the object loaded by getObject().

E.g. for image:

def __loadedHotSwap__(self):
        ctx = self._conn.SERVICE_OPTS.copy()
        ctx.setOmeroGroup(self.getDetails().group.id.val)
        self._obj = self._conn.getContainerService().getImages(
            self.OMERO_CLASS, (self._oid,), None, ctx)[0]

To test:

Set some values for ExternalInfo:
Using this branch, we can set EntityId, EntityType (both required) and Lsid, UUid (optional):

$ omero obj ext-info-set Image:1 3 mytype mylsid myuuid

Since the JSON API uses _getQueryString() to build queries, and omero-marshal already handles externalInfo, you can check the details at E.g. /api/v0/m/images/ID/: (NB: deployed on idr-testing, e.g. /api/v0/m/images/15159664/).

"externalInfo": {
    "@type": "TBD#ExternalInfo",
    "@id": 8,
    "omero:details": {},
    "EntityId": 3,
    "EntityType": "com.glencoesoftware.ngff:multiscales",
    "Lsid": "test_dataset_extinfo"
}

will-moore · 2025-03-21T15:45:52Z

@jburel I added support for:

$ omero obj ext-info-set Image:12 myLsid myEntityType

so we can set externalInfo on any object now.

snoopycrimecop · 2025-03-22T00:39:48Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#350. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-23T00:38:26Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#351. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-24T00:43:56Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#352. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-25T01:22:33Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#353. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-26T00:45:46Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#354. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-27T00:40:54Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#355. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-28T07:12:31Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#356. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

--conflicts

snoopycrimecop · 2025-03-28T07:27:51Z

Conflicting PR. Removed from build OMERO-python-superbuild-push#357. See the console output for more details.
Possible conflicts:

PR Add parents and children to omero obj #299 joshmoore 'Add parents and children to omero obj'
- src/omero/plugins/obj.py

~~--conflicts~~ Conflict resolved in build OMERO-python-superbuild-push#358. See the console output for more details.

dominikl

👍 Looks good to me.Thanks Will, useful command!

sbesson

The changes to omero.gateway to fetch the externalinfo is self-contained and allows this metadata to be exposed in the OMERO.web client as well as in the OMERO.web API.
Performance-wise, I understand we have seen no degradation although it might be worth testing in the context of a graph like a Plate?

The new command is an interesting starting point to be able to interact with these objects in command line. The current implementation raises a few design questions listed below which should be addressed before this can be merged. If it was valuable to get the gateway changes in sooner, this could be split into two separate PRs.

as this PR allow to set the external info associated with any object, do we need/want a mirroring functionality to get it? One possibility would be to leverage the omero obj get functionality:

 % ./venv/bin/omero obj get ExternalInfo:799
Using session for [email protected]:4064. Idle timeout: 10 min. 
Current group: Public
entityId=3
entityType=com.glencoesoftware.ngff:multiscales
id=799
lsid=/bia-idr/S-BIAD1483/Experiment/P30/WT/Animal_1/Image_18.ome.zarr/media/taz/DATA2/Final_Tif/P30/WT/Animal_1/Image_18.ome.zarr/0
uuid=

In that case, the missing link is to retrieve the ID of the ExternalInfo associated with any object. Could this be another output of omero obj get?

ExternalInfo is immutable so calling omero ext-info-set <Object>:<id> with different properties will not update the attributes. Either the command needs to handle this or it should fail with an informative error message suggesting to delete the existing external info first (via omero delete)
given ExternalInfo is an object with defined attribute names unlike map, should the command use the same semantics as omero obj update or omero obj new i.e. omero obj ext-info-set <Object>:<id> <field>=<value> [<field>=<value> ...]?

will-moore · 2025-05-21T17:15:49Z

@sbesson Thanks for the review.

Re: performance - for a Plate that contains lots of other objects, the ExternalInfo is only loaded for the root Plate object, not all the Wells, Images etc. So, performance is not an issue. But I guess this raises the question: "should we load ExternalInfo" for EVERY object that is loaded in the graph? This would be a lot more work and I'm not sure if it's needed? In the various graph-loading queries we have, we make a judgement call on how much of the graph to load, and this is just another of those questions.

For getting the extInfo, how about we support:

omero obj ext-info-get Image:12
entityId=3
entityType=myEntityType
id=101
lsid=myLsid
uuid=

The pair of new methods omero obj ext-info-set and omero obj ext-info-get would correspond to existing omero obj map-set and omero obj map-get.

I guess we could also support getting individual properties:

omero obj ext-info-get Image:12 lsid
myLsid

Calling omero ext-info-set : with different properties won't try to update the immutable ExternalInfo object, but will create a new ExternalInfo with those updated properties.

I'm not sure we need to support ext-info-set with field values. It just seems such an edge case. Mostly we want to update both lsid and entityType together, which you can do as above.

If this looks good, I'll implement ext-info-get?

sbesson · 2025-05-22T10:51:12Z

Implementing some equivalent get method would certainly help assessing this PR.

Calling omero ext-info-set : with different properties won't try to update the immutable ExternalInfo object, but will create a new ExternalInfo with those updated properties.

You're correct. What confused me during the initial code review is that extinfo is used for the retrieved the exiting ExternalInfoif applicable and later recycled for creating a new object.
What is the rationale/workflow for copying properties from an existing ExternalInfo object? To reduce the number of cases to consider, would it be simpler to only consider the values passed in command-line. In that case, both entityType and entityId should be mandatory with lsid (and uuid) optional.

will-moore · 2025-05-26T11:16:35Z

@sbesson I think I was influenced by ome/omero-cli-zarr#167 which allows setting of just the entityType or entityId, but this is probably not needed and it's simpler to only allow setting of both together (which is what I've done in all my usage of this PR so far). I'll update to that...

will-moore · 2025-05-26T11:49:06Z

@sbesson I'm actually not sure what should be mandatory / optional. I agree that uuid is optional, but NOT lsid?
Is there any use for ExternalInfo if it doesn't have lsid?
e.g.

entityId=3
entityType="com.glencoesoftware.ngff:multiscales"
lsid="/bia-idr/S-BIAD1483/Experiment/P30/WT/Animal_1/Image_18.ome.zarr"
uuid=

I don't know about entityId (what does this even mean). Do we want users to specify that it's 3? Or can we use 3 as a default?

I'm kinda assuming you got these attributes mixed up above "In that case, both entityType and entityId should be mandatory with lsid (and uuid) optional." ??

sbesson · 2025-05-26T12:40:51Z

I'm actually not sure what should be mandatory / optional. I agree that uuid is optional, but NOT lsid?

My suggestion was based on the omero-model mappings which translated into the database schema where only entityid and entitytype are not nullable

omero=> \d externalinfo;

                      Table "public.externalinfo"
   Column    |          Type          | Collation | Nullable | Default 
-------------+------------------------+-----------+----------+---------
 id          | bigint                 |           | not null | 
 permissions | bigint                 |           | not null | 
 entityid    | bigint                 |           | not null | 
 entitytype  | character varying(255) |           | not null | 
 lsid        | text                   |           |          | 
 uuid        | character varying(255) |           |          | 
 creation_id | bigint                 |           | not null | 
 external_id | bigint                 |           |          | 
 group_id    | bigint                 |           | not null | 
 owner_id    | bigint                 |           | not null |

Is there any use for ExternalInfo if it doesn't have lsid?

In the case of omero-zarr-pixel-buffer extension, I agree no lsid would be meaningless. In general, someone could construct an extension point that only requires entitytype/entityid to be present.

I don't know about entityId (what does this even mean). Do we want users to specify that it's 3? Or can we use 3 as a default?

No we cannot assume that 3 should be the default. In the case of omero-zarr-pixel-buffer, the entityId field is really used as to communicate this is the third version of the specification to store OME-Zarr label/images in the OMERO database (the two first versions were internal only).

will-moore · 2025-05-26T13:12:04Z

OK, understood.
Supported now:

$ omero obj ext-info-set Project:302
usage: ext-info-set OBJ entityId entityType [lsid] [uuid]

$omero obj ext-info-set Project:302 3 myEntType
Project:302

$ omero obj ext-info-get Project:302
id=108
entityId=3
entityType=myEntType
lsid=
uuid=

$ omero obj ext-info-set Project:302 3 myEntityType2 this.is.my.lsid
Project:302

$ omero obj ext-info-get Project:302
id=109
entityId=3
entityType=myEntityType2
lsid=this.is.my.lsid
uuid=

$ omero obj ext-info-get Project:302 lsid
this.is.my.lsid

$ omero obj ext-info-get Project:302 uuid


$ omero obj ext-info-get Project:302 uuid_foo
usage: ext-info-get OBJ [id|lsid|entityId|entityType|uuid]

$ omero obj ext-info-get Project:301
No ExternalInfo found: Project:301

sbesson

With the last changes, getting and setting the external info on an object works as expected.

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj new Project name=test
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Project:1
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
No ExternalInfo found: Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
usage: ext-info-set OBJ entityId entityType [lsid] [uuid]
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1 0 mytype
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Project:1         
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=1
entityId=0
entityType=mytype
lsid=
uuid=

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1 0 mytype mylsid
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Project:1                
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=2
entityId=0
entityType=mytype
lsid=mylsid
uuid=

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1 0 mytype mylsid myuuid
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Project:1                       
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=3
entityId=0
entityType=mytype
lsid=mylsid
uuid=myuuid

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1 0 mytype              
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Project:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Project:1          
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=4
entityId=0
entityType=mytype
lsid=
uuid=

Two minor issues found during testing:

the sequence of testing above results in a certain number of orphaned objects. In the case of omero obj ext-info-set, should the command also delete previous external info after the update service call if applicable?

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero hql 'select e from 
ExternalInfo e' -q
 # | Class         | Id | details         | entityId | entityType | lsid   | uuid   
---+---------------+----+-----------------+----------+------------+--------+--------
 0 | ExternalInfoI | 1  | owner=0;group=0 | 0        | mytype     |        |        
 1 | ExternalInfoI | 2  | owner=0;group=0 | 0        | mytype     | mylsid |        
 2 | ExternalInfoI | 3  | owner=0;group=0 | 0        | mytype     | mylsid | myuuid 
 3 | ExternalInfoI | 4  | owner=0;group=0 | 0        | mytype     |        |        
(4 rows)

passing an invalid entityId fails with a non-user friendly error message. Could this be caught when casting the input to int?

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Project:1 invalidid mytype 
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Traceback (most recent call last):
  File "/Users/sbesson/Documents/GitHub/omero-py/./venv/bin/omero", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sbesson/Documents/GitHub/omero-py/venv/lib/python3.12/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
         ^^^^^^^^^^^^^^^^
  File "/Users/sbesson/Documents/GitHub/omero-py/venv/lib/python3.12/site-packages/omero/cli.py", line 1771, in argv
    cli.invoke(args[1:])
  File "/Users/sbesson/Documents/GitHub/omero-py/venv/lib/python3.12/site-packages/omero/cli.py", line 1208, in invoke
    stop = self.onecmd(line, previous_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sbesson/Documents/GitHub/omero-py/venv/lib/python3.12/site-packages/omero/cli.py", line 1285, in onecmd
    self.execute(line, previous_args)
  File "/Users/sbesson/Documents/GitHub/omero-py/venv/lib/python3.12/site-packages/omero/cli.py", line 1367, in execute
    args.func(args)
  File "<string>", line 652, in process
  File "<string>", line 268, in go
  File "<string>", line 289, in on_go
ValueError: invalid literal for int() with base 10: 'invalidid'

will-moore · 2025-05-29T18:48:31Z

@sbesson Thanks - addressed those 2 points

sbesson

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Image:1
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
No ExternalInfo found: Image:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Image:1 invalid mytpe
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
entityId must be an integer, got: invalid
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Image:1 0 mytype     
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Image:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Image:1              
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=1
entityId=0
entityType=mytype
lsid=
uuid=

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Image:1 1 mytype
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Image:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Image:1         
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=2
entityId=1
entityType=mytype
lsid=
uuid=

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-set Image:1 1 mytype mylsid myuuid
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
Image:1
sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero obj ext-info-get Image:1                       
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
id=3
entityId=1
entityType=mytype
lsid=mylsid
uuid=myuuid

sbesson@Sebastiens-MacBook-Pro-3 omero-py % ./venv/bin/omero hql 'select e from ExternalInfo e'         
Using session for root@localhost:4064. Idle timeout: 10 min. Current group: system
 # | Class         | Id | details         | entityId | entityType | lsid   | uuid   
---+---------------+----+-----------------+----------+------------+--------+--------
 0 | ExternalInfoI | 3  | owner=0;group=0 | 1        | mytype     | mylsid | myuuid 
(1 row)

To see details for object, enter line number.
To move ahead one page, enter 'p'
To re-display list, enter 'r'.
To quit, enter 'q' or just enter.
q

imagesc-bot · 2025-06-16T12:58:51Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/compatibility-of-ome-zarr-stored-on-s3-and-omero/111380/17

will-moore force-pushed the externalInfo branch from 559333d to 40028fa Compare March 13, 2025 15:40

getObject() and getExternalInfo() load externalInfo

295d948

will-moore force-pushed the externalInfo branch from 40028fa to 295d948 Compare March 13, 2025 15:41

This was referenced Mar 13, 2025

externalInfo display ome/omero-web#613

Open

getQueryString includes externalInfo #449

Closed

omero obj ext-info-set Image:12 myLsid myEntityType

ff9ad44

sbesson mentioned this pull request Mar 28, 2025

Add parents and children to omero obj #299

Open

will-moore requested a review from dominikl April 15, 2025 14:45

dominikl approved these changes Apr 16, 2025

View reviewed changes

sbesson requested changes Apr 21, 2025

View reviewed changes

will-moore added 2 commits May 26, 2025 12:52

obj ext-info-set MUST specify lsid AND entityType

3ed7582

Support omero obj ext-info-get Image:1

9d90d09

lsid and uuid optional, entityId and entityType required

7ccbada

Fix py3.9 syntax error on f-string

dd2acb2

sbesson self-requested a review May 26, 2025 13:31

sbesson requested changes May 29, 2025

View reviewed changes

Handle invalid entity_id. Delete orphaned ExternalInfo

41a85bd

sbesson approved these changes May 30, 2025

View reviewed changes

sbesson merged commit d2f8480 into ome:master May 30, 2025
14 checks passed

will-moore mentioned this pull request Sep 16, 2025

README updates (requirements, installation & usage) glencoesoftware/omero-zarr-pixel-buffer#35

Merged

This was referenced Sep 30, 2025

Python gateway: ExternalInfo getter regression #481

Closed

omero.gateway: fix DetailsWrapper.getExternalInfo regression #482

Merged

omero.gateway: deprecate BlitzObjectWrapper.getExternalInfo() #483

Merged

sbesson added this to OMERO.server 5.6.17 Jan 11, 2026

github-project-automation bot moved this to Todo in OMERO.server 5.6.17 Jan 11, 2026

github-project-automation bot moved this from Todo to Done in OMERO.server 5.6.17 Jan 11, 2026

getObject() and getExternalInfo() load externalInfo #453

getObject() and getExternalInfo() load externalInfo #453

Uh oh!

Conversation

will-moore commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

will-moore commented Mar 21, 2025

Uh oh!

snoopycrimecop commented Mar 22, 2025

Uh oh!

snoopycrimecop commented Mar 23, 2025

Uh oh!

snoopycrimecop commented Mar 24, 2025

Uh oh!

snoopycrimecop commented Mar 25, 2025

Uh oh!

snoopycrimecop commented Mar 26, 2025

Uh oh!

snoopycrimecop commented Mar 27, 2025

Uh oh!

snoopycrimecop commented Mar 28, 2025

Uh oh!

snoopycrimecop commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dominikl left a comment

Choose a reason for hiding this comment

Uh oh!

sbesson left a comment

Choose a reason for hiding this comment

Uh oh!

will-moore commented May 21, 2025

Uh oh!

sbesson commented May 22, 2025

Uh oh!

will-moore commented May 26, 2025

Uh oh!

will-moore commented May 26, 2025

Uh oh!

sbesson commented May 26, 2025

Uh oh!

will-moore commented May 26, 2025

Uh oh!

sbesson left a comment

Choose a reason for hiding this comment

Uh oh!

will-moore commented May 29, 2025

Uh oh!

sbesson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

imagesc-bot commented Jun 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

will-moore commented Mar 13, 2025 •

edited

Loading

snoopycrimecop commented Mar 28, 2025 •

edited

Loading