Add ZMQ API message for requesting the current state of the RunEngine Metadata

[jwlodek]

Description

The Run Engine metadata is often used to store important state about the currently running experiment. For example, we use it to store the current cycle and proposal number as well as the current transient scan ID. Beamlines have requested having the ability to see this information from an interface to QueueServer (specifically queue-monitor). The new API message allows for asking for the current state of the metadata provided the environment is open.

Motivation and Context

Beamlines wish to see the value of certain metadata keys in the UI the use for queueserver (eg. transient scan id, proposal number etc). Subsequent PRs can add the ability to use this message in the python and http apis, and eventually queuemonitor and other queueserver clients.

Summary of Changes for Release Notes

Added

• New handlers for processing the re_metadata message to the RE manager and RE worker classes.
• New qserver re metadata CLI command.
• Unit test for the handlers and API.

Changed

• Unit test for the CLI to include the new command.
• Example profile to add a sample key/value to the RunEngine metadata to start.

How Has This Been Tested?

Manually tested:

(venv) $ qserver re metadata
Arguments: ['re', 'metadata']
13:26:55 - MESSAGE: 
{'success': False,
 'msg': 'Environment does not exist. Cannot retrieve Run Engine metadata.',
 're_metadata': {}}
(venv) $ qserver environment open
Arguments: ['environment', 'open']
13:27:04 - MESSAGE: 
{'success': True, 'msg': ''}
(venv) $ qserver re metadata
Arguments: ['re', 'metadata']
13:27:10 - MESSAGE: 
{'success': True,
 'msg': '',
 're_metadata': {'metadata_key': 'metadata_value',
                 'versions': {'ophyd': '1.11.0',
                              'ophyd_async': '0.14.2',
                              'bluesky': '1.14.6'}}}

And added in unit tests.

[dmgav]

Another point worth investigation is how the server behaves if RE metadata dictionary is not serializable. Ideally the request should fail with meaningful error message.

[jwlodek]

I am not sure if there's a better way to check for serializability other than just trying to serialize?

[dmgav]

This is what I did for the return value of an arbitrary function, which can return arbitrary result:

bluesky-queueserver/bluesky_queueserver/manager/worker.py

Line 551 in 934008c

json.dumps(return_value) # The result of the conversion is intentionally discarded

I believe the message is pickled when it is passed between processes, so the results of JSON serialization are not used.

[jwlodek]

@dmgav I don't think the test failures are related to the changes here. Is this OK to merge or is there something else you wanted me to address first?

[dmgav]

We need to handle the situations when there is no RE in the environment, RE does not have md attribute or RE.md can not be converted to a dictionary. Probably the request should fail with useful message if there is no RE in the environment and return {} in the other cases. For testing, RE can be removed from the existing environment or replaced by executing a script, e.g.:

    resp2, _ = zmq_request("script_upload", params={"script": "delete RE"})
    assert resp2["success"] is True
    result = wait_for_task_result(time=10, task_uid=resp2["task_uid"])
    assert result["success"] is True, pprint.pformat(result)

To check that RE was removed successfully it may be useful to check re_state field of the status (it should be None).

[dmgav]

Since we are adding a new API, we should also update the API documentation, may be put the API description after the description of re-runs API : https://blueskyproject.io/bluesky-queueserver/re_manager_api.html#re-runs