Skip to content

Commit 6536118

Browse files
DiamondJosephkeithralphs
DiamondJoseph
and
keithralphs
authored
feat: Allow automatic insertion of documents into Tiled (#755)
Co-authored-by: Keith Ralphs <keith.ralphs@diamond.ac.uk>
1 parent b8b2adf commit 6536118

15 files changed

Lines changed: 219 additions & 17 deletions

File tree

‎.github/workflows/_system_test.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ on:
44
env:
55
# https://github.com/pytest-dev/pytest/issues/2042
66
PY_IGNORE_IMPORTMISMATCH: "1"
7+
TILED_SINGLE_USER_API_KEY: unknown # ${TILED_SINGLE_USER_API_KEY}
78

89
jobs:
910
run:
@@ -42,6 +43,7 @@ jobs:
4243
services: |
4344
rabbitmq
4445
numtracker
46+
tiled
4547
4648
- name: Start Blueapi Server
4749
env:

‎dev-requirements.txt‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,20 @@ annotated-types==0.7.0
99
anyio==4.9.0
1010
asgiref==3.9.1
1111
attrs==25.3.0
12+
awkward==2.8.5
13+
awkward_cpp==47
1214
babel==2.17.0
1315
beautifulsoup4==4.13.4
1416
bidict==0.23.1
17+
blosc2==3.6.1
1518
bluesky==1.14.2
1619
bluesky-stomp==0.1.7
1720
certifi==2025.7.14
1821
cffi==2.0.0
1922
cfgv==3.4.0
2023
charset-normalizer==3.4.2
2124
click==8.2.1
25+
cloudpickle==3.1.1
2226
colorama==0.4.6
2327
colorlog==6.9.0
2428
compress-pickle==2.1.0
@@ -27,6 +31,7 @@ copier==9.9.1
2731
coverage==7.10.6
2832
cryptography==46.0.2
2933
cycler==0.12.1
34+
dask==2025.7.0
3035
dataclasses-json==0.6.7
3136
deepdiff==8.6.1
3237
deepmerge==2.0
@@ -47,6 +52,7 @@ flexcache==0.3
4752
flexparser==0.4
4853
fonttools==4.59.0
4954
frozenlist==1.7.0
55+
fsspec==2025.7.0
5056
funcy==2.0
5157
gitdb==4.0.12
5258
GitPython==3.1.45
@@ -67,10 +73,15 @@ iniconfig==2.1.0
6773
itsdangerous==2.2.0
6874
Jinja2==3.1.6
6975
jinja2-ansible-filters==1.3.2
76+
json-merge-patch==0.3.0
77+
jsonpatch==1.33
78+
jsonpointer==3.0.0
7079
jsonschema==4.25.0
7180
jsonschema-specifications==2025.4.1
7281
jwcrypto==1.5.6
7382
kiwisolver==1.4.8
83+
llvmlite==0.44.0
84+
locket==1.0.0
7485
lz4==4.4.4
7586
markdown-it-py==3.0.0
7687
MarkupSafe==3.0.2
@@ -85,9 +96,12 @@ msgpack-numpy==0.4.8
8596
multidict==6.6.3
8697
mypy_extensions==1.1.0
8798
myst-parser==4.0.1
99+
ndindex==1.10.0
88100
networkx==3.5
89101
nodeenv==1.9.1
90102
nose2==0.15.1
103+
numba==0.61.2
104+
numexpr==2.11.0
91105
numpy==2.2.6
92106
observability-utils==0.1.5
93107
opencv-python-headless==4.12.0.88
@@ -110,6 +124,8 @@ orderly-set==5.5.0
110124
orjson==3.11.1
111125
p4p==4.2.1
112126
packaging==25.0
127+
pandas==2.3.1
128+
partd==1.4.2
113129
pathlib2==2.3.7.post1
114130
pathspec==0.12.1
115131
picobox==4.0.0
@@ -127,6 +143,8 @@ propcache==0.3.2
127143
protobuf==6.31.1
128144
pvxslibs==1.4.0
129145
py==1.11.0
146+
py-cpuinfo==9.0.0
147+
pyarrow==21.0.0
130148
pycparser==2.22
131149
pydantic==2.11.7
132150
pydantic-extra-types==2.10.5
@@ -145,6 +163,7 @@ pytest-cov==7.0.0
145163
python-dateutil==2.9.0.post0
146164
python-dotenv==1.1.1
147165
python-multipart==0.0.20
166+
pytz==2025.2
148167
PyYAML==6.0.2
149168
pyzmq==26.3.0
150169
questionary==2.1.0
@@ -170,6 +189,7 @@ smmap==5.0.2
170189
sniffio==1.3.1
171190
snowballstemmer==3.0.1
172191
soupsieve==2.7
192+
sparse==0.17.0
173193
Sphinx==8.2.3
174194
sphinx-autobuild==2025.8.25
175195
sphinx-click==6.0.0
@@ -189,6 +209,7 @@ starlette==0.47.2
189209
stomp.py==8.2.0
190210
super-state-machine==2.0.2
191211
tenacity==9.1.2
212+
tiled==0.1.0b30
192213
tomlkit==0.13.3
193214
toolz==1.0.0
194215
tox==3.28.0
@@ -202,6 +223,7 @@ types-urllib3==1.26.25.14
202223
typing-inspect==0.9.0
203224
typing-inspection==0.4.1
204225
typing_extensions==4.14.1
226+
tzdata==2025.2
205227
ujson==5.10.0
206228
urllib3==2.5.0
207229
uvicorn==0.35.0
@@ -214,6 +236,8 @@ websocket-client==1.8.0
214236
websockets==15.0.1
215237
workflows==3.2
216238
wrapt==1.17.2
239+
xarray==2025.7.1
217240
yarl==1.20.1
218241
zipp==3.23.0
219242
zocalo==1.3.0
243+
zstandard==0.23.0

‎helm/blueapi/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,4 +51,4 @@ A Helm chart deploying a worker pod that runs Bluesky plans
5151
| worker.env.sources | list | `[{"kind":"planFunctions","module":"dodal.plans"},{"kind":"planFunctions","module":"dodal.plan_stubs.wrapped"}]` | modules (must be installed in the venv) to fetch devices/plans from |
5252
| worker.logging | object | `{"graylog":{"enabled":false,"url":"tcp://graylog-log-target.diamond.ac.uk:12231/"},"level":"INFO"}` | Configures logging. Port 12231 is the `dodal` input on graylog which will be renamed `blueapi` |
5353
| worker.scratch | object | `{"repositories":[],"root":"/blueapi-plugins/scratch"}` | If initContainer is enabled the default branch of python projects in this section are installed into the venv *without their dependencies* |
54-
| worker.stomp | object | `{"auth":{"password":"guest","username":"guest"},"enabled":false,"url":"tcp://rabbitmq:61613/"}` | Message bus configuration for returning status to GDA/forwarding documents downstream Password may be in the form ${ENV_VAR} to be fetched from an environment variable e.g. mounted from a SealedSecret |
54+
| worker.stomp | object | `{"auth":{"password":"guest","username":"guest"},"enabled":false,"url":"tcp://rabbitmq:61613/"}` | Message bus configuration for returning status to GDA/forwarding documents downstream Password may be in the form ${ENV_VAR} to be fetched from an environment variable e.g. mounted from a SealedSecret |

‎helm/blueapi/config_schema.json‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@
167167
"additionalProperties": false,
168168
"properties": {
169169
"url": {
170-
"default": "http://localhost:8002/graphql",
170+
"default": "http://localhost:8406/graphql",
171171
"format": "uri",
172172
"maxLength": 2083,
173173
"minLength": 1,
@@ -359,6 +359,39 @@
359359
"title": "StompConfig",
360360
"type": "object"
361361
},
362+
"TiledConfig": {
363+
"additionalProperties": false,
364+
"properties": {
365+
"enabled": {
366+
"default": false,
367+
"description": "True if blueapi should forward data to a Tiled instance",
368+
"title": "Enabled",
369+
"type": "boolean"
370+
},
371+
"url": {
372+
"default": "http://localhost:8407/",
373+
"format": "uri",
374+
"maxLength": 2083,
375+
"minLength": 1,
376+
"title": "Url",
377+
"type": "string"
378+
},
379+
"api_key": {
380+
"anyOf": [
381+
{
382+
"type": "string"
383+
},
384+
{
385+
"type": "null"
386+
}
387+
],
388+
"default": null,
389+
"title": "Api Key"
390+
}
391+
},
392+
"title": "TiledConfig",
393+
"type": "object"
394+
},
362395
"WorkerEventConfig": {
363396
"additionalProperties": false,
364397
"description": "Config for event broadcasting via the message bus",
@@ -379,6 +412,9 @@
379412
"stomp": {
380413
"$ref": "#/$defs/StompConfig"
381414
},
415+
"tiled": {
416+
"$ref": "#/$defs/TiledConfig"
417+
},
382418
"env": {
383419
"$ref": "#/$defs/EnvironmentConfig"
384420
},

‎helm/blueapi/values.schema.json‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,6 +482,9 @@
482482
},
483483
"stomp": {
484484
"$ref": "#/$defs/StompConfig"
485+
},
486+
"tiled": {
487+
"$ref": "#/$defs/TiledConfig"
485488
}
486489
},
487490
"additionalProperties": false,
@@ -653,7 +656,7 @@
653656
"properties": {
654657
"url": {
655658
"title": "Url",
656-
"default": "http://localhost:8002/graphql",
659+
"default": "http://localhost:8406/graphql",
657660
"type": "string",
658661
"maxLength": 2083,
659662
"minLength": 1
@@ -835,6 +838,37 @@
835838
},
836839
"additionalProperties": false
837840
},
841+
"TiledConfig": {
842+
"title": "TiledConfig",
843+
"type": "object",
844+
"properties": {
845+
"api_key": {
846+
"title": "Api Key",
847+
"anyOf": [
848+
{
849+
"type": "string"
850+
},
851+
{
852+
"type": "null"
853+
}
854+
]
855+
},
856+
"enabled": {
857+
"title": "Enabled",
858+
"description": "True if blueapi should forward data to a Tiled instance",
859+
"default": false,
860+
"type": "boolean"
861+
},
862+
"url": {
863+
"title": "Url",
864+
"default": "http://localhost:8407/",
865+
"type": "string",
866+
"maxLength": 2083,
867+
"minLength": 1
868+
}
869+
},
870+
"additionalProperties": false
871+
},
838872
"WorkerEventConfig": {
839873
"title": "WorkerEventConfig",
840874
"description": "Config for event broadcasting via the message bus",

‎helm/blueapi/values.yaml‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ podAnnotations: {}
3636
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
3737
podLabels: {}
3838

39-
podSecurityContext: {}
39+
podSecurityContext:
40+
{}
4041
# fsGroup: 2000
4142

4243
securityContext:
@@ -61,7 +62,8 @@ service:
6162
ingress:
6263
enabled: false
6364
className: "nginx"
64-
annotations: {}
65+
annotations:
66+
{}
6567
# kubernetes.io/ingress.class: nginx
6668
# kubernetes.io/tls-acme: "true"
6769
hosts:
@@ -129,18 +131,16 @@ startupProbe:
129131
failureThreshold: 5
130132
periodSeconds: 10
131133

132-
133-
134134
# -- Additional volumes on the output StatefulSet definition.
135135
# Define volumes from e.g. Secrets, ConfigMaps or the Filesystem
136136
volumes: []
137137

138138
# -- Additional volumeMounts on the output StatefulSet definition.
139139
# Define how volumes are mounted to the container referenced by using the same name.
140140
volumeMounts:
141-
- name: worker-config
142-
mountPath: "/config"
143-
readOnly: true
141+
- name: worker-config
142+
mountPath: "/config"
143+
readOnly: true
144144

145145
# -- May be required to run on specific nodes (e.g. the control machine)
146146
nodeSelector: {}
@@ -152,18 +152,18 @@ affinity: {}
152152
# blueapi specific fields
153153

154154
# -- May be needed for EPICS depending on gateway configuration
155-
hostNetwork: false
155+
hostNetwork: false
156156

157157
# -- If enabled the blueapi pod will restart on changes to `worker`
158158
restartOnConfigChange: true
159159

160160
# -- Additional envVars to mount to the pod
161161
extraEnvVars: []
162-
# - name: RABBITMQ_PASSWORD
162+
# - name: TILED_API_KEY
163163
# valueFrom:
164164
# secretKeyRef:
165-
# name: rabbitmq-password
166-
# key: rabbitmq-password
165+
# name: tiled_api_key
166+
# key: tiled_api_key
167167

168168
# -- Configure tracing: opentelemetry-collector.tracing should be available in all Diamond clusters
169169
tracing:
@@ -191,7 +191,7 @@ worker:
191191
- kind: planFunctions
192192
module: dodal.plan_stubs.wrapped
193193
# -- Message bus configuration for returning status to GDA/forwarding documents downstream
194-
# Password may be in the form ${ENV_VAR} to be fetched from an environment variable e.g. mounted from a SealedSecret
194+
# Password may be in the form ${ENV_VAR} to be fetched from an environment variable e.g. mounted from a SealedSecret
195195
stomp:
196196
enabled: false
197197
auth:

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ classifiers = [
1212
]
1313
description = "Lightweight bluesky-as-a-service wrapper application. Also usable as a library."
1414
dependencies = [
15+
"tiled[client]",
1516
"bluesky[plotting]>=1.13.1", # plotting includes matplotlib, required for BestEffortCallback in run plans
1617
"ophyd-async",
1718
"aioca",

‎src/blueapi/config.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,15 @@ class StompConfig(BlueapiBaseModel):
7575
)
7676

7777

78+
class TiledConfig(BlueapiBaseModel):
79+
enabled: bool = Field(
80+
description="True if blueapi should forward data to a Tiled instance",
81+
default=False,
82+
)
83+
url: HttpUrl = HttpUrl("http://localhost:8407")
84+
api_key: str | None = os.environ.get("TILED_SINGLE_USER_API_KEY", None)
85+
86+
7887
class WorkerEventConfig(BlueapiBaseModel):
7988
"""
8089
Config for event broadcasting via the message bus
@@ -211,7 +220,7 @@ def id_token_signing_alg_values_supported(self) -> list[str]:
211220

212221

213222
class NumtrackerConfig(BlueapiBaseModel):
214-
url: HttpUrl = HttpUrl("http://localhost:8002/graphql")
223+
url: HttpUrl = HttpUrl("http://localhost:8406/graphql")
215224

216225

217226
class ApplicationConfig(BlueapiBaseModel):
@@ -221,6 +230,7 @@ class ApplicationConfig(BlueapiBaseModel):
221230
"""
222231

223232
stomp: StompConfig = Field(default_factory=StompConfig)
233+
tiled: TiledConfig = Field(default_factory=TiledConfig)
224234
env: EnvironmentConfig = Field(default_factory=EnvironmentConfig)
225235
logging: LoggingConfig = Field(default_factory=LoggingConfig)
226236
api: RestConfig = Field(default_factory=RestConfig)

0 commit comments

Comments
 (0)