Models: Remove dependency on API for sending results?

[wk9874]

Is your feature request related to a problem? Please describe.
It probably isn't ideal to have Ray having to do API requests to post samples / annotations since Ray must already have an open connection to the head node with db access...
Describe the solution you'd like
Could change this to have a long running process which monitors any tasks added to the TaskRegistry, doesd .wait() to see if they are done, and if they are get the results and add them to the db. This would also fix the issue with the unit tests we have currently, where after the removal of local mode the ray worker cant ping the API.

[wk9874]

I'm in two minds about this, maybe worth a discussion with Sam when he's back... Changing it would require adding a class which is constantly polling any tasks inside the ActorRegistry waiting to see if they are done, and add those to the db when they appear... Maybe this is preferable to the API method? Not sure what the standard way to handle this in Ray is

For now I've done something similar inside the models tests to get them working - basically the ray worker will not try to send anything to the API (it cant, the HttpxClient is not available to the worker thread) because the API_URL env var is set to None. The test will then get() the result of the task scheduled on the UI, and add the results to the db over the test API instead.

[wk9874]

It would also make sending updates during training more difficult, since those dont arrive nicely at the end of a task, but mid way through

Would have to still send model updates via API, just final model status and model predict samples/annotations via polling on backend. So there would be two different ways its writing stuff to db