Tiled uses Websockets to provide notifications to clients when datasets are created, updated, written to, or "closed". The clients must maintain active Websocket connections to the server in order to receive updates. If a client becomes disconnected, it is the client's responsibility to track their last update and either ask the server to "replay" recent updates that it missed (within limits) or use the REST API to load data from storage at rest. This "best effort" quality of service enables Tiled to service a potentially large number of clients, adapting its level of service (e.g., how much history is retained for replay) to fit within bounded RAM under varying load.
Webhooks might be a better fit for use cases such as kicking off workflow jobs when datasets are created or closed. Here, we want a stronger guarantee of delivery. The updates are less granular and therefore impose less burden on the server: we want a notification when datasets are created or closed, but not on every incremental write.
Within a workflow job, the Websockets API or REST API may be used to access data, depending on the requirements of the workflow. But Webhooks may be a better fit than Websockets for kicking off the jobs themselves. Webhooks can implement retry to bridge over a brief outage in the consumer, and it can provide a history of success/failure to facilitate outage recovery. Webhooks do not require subscribers to hold active open connections to receive updates.
API sketch:
-
An authorized client can register a webhook that fires when a child of node /x is created. The POST body should include a URL that Tiled will be POST to. The server will store this, probably in a new PG table webhooks, which a foreign key to the nodes table.
POST /api/v1/webhook/target/x
-
The client can review webhooks registered on x.
GET /api/v1/webhook/target/x
-
Now say support a client creates a child of x.
POST /api/v1/metadata/x
During the processing of request (3):
- JSON is stored in Redis and broadcast to any WS clients (as now)
- JSON is stored in PG and becomes available via REST API (as now)
- This event is logged in some kind of database of recent webhook events: probably a separate Redis db with a generous but finite TTL, or maybe PG.
- A starlette
BackgroundTask is scheduled, which will send the webhook POST. This includes a reasonable but not unlimited number of retries. The outcome—success or failure, number of attemps—is added to the log mentioned above.
- For debugging or outage recovery, client can review (recent) history of webhook invocations
GET /api/v1/webhook/history/{webhook_id}
Tiled uses Websockets to provide notifications to clients when datasets are created, updated, written to, or "closed". The clients must maintain active Websocket connections to the server in order to receive updates. If a client becomes disconnected, it is the client's responsibility to track their last update and either ask the server to "replay" recent updates that it missed (within limits) or use the REST API to load data from storage at rest. This "best effort" quality of service enables Tiled to service a potentially large number of clients, adapting its level of service (e.g., how much history is retained for replay) to fit within bounded RAM under varying load.
Webhooks might be a better fit for use cases such as kicking off workflow jobs when datasets are created or closed. Here, we want a stronger guarantee of delivery. The updates are less granular and therefore impose less burden on the server: we want a notification when datasets are created or closed, but not on every incremental write.
Within a workflow job, the Websockets API or REST API may be used to access data, depending on the requirements of the workflow. But Webhooks may be a better fit than Websockets for kicking off the jobs themselves. Webhooks can implement retry to bridge over a brief outage in the consumer, and it can provide a history of success/failure to facilitate outage recovery. Webhooks do not require subscribers to hold active open connections to receive updates.
API sketch:
An authorized client can register a webhook that fires when a child of node
/xis created. The POST body should include a URL that Tiled will be POST to. The server will store this, probably in a new PG tablewebhooks, which a foreign key to thenodestable.POST /api/v1/webhook/target/xThe client can review webhooks registered on x.
GET /api/v1/webhook/target/xNow say support a client creates a child of x.
POST /api/v1/metadata/xDuring the processing of request (3):
BackgroundTaskis scheduled, which will send the webhook POST. This includes a reasonable but not unlimited number of retries. The outcome—success or failure, number of attemps—is added to the log mentioned above.GET /api/v1/webhook/history/{webhook_id}