Right now the supervisor export script sets a sentinel flag file telling the web server to return status 512, drains the results table into a text file, removes the flag file, sorts the text file, formats, and then uploads. During this time, the web service is unavailable for 30 minutes.
Instead, it could slowly drain the results table into another database to avoid blocking the web server and then continue from there. It would also provide some way of resuming in case of failure.
Edit: It could also just check for the current export json file, drain into appending the text file, but allow resuming in case of failure.
Right now the supervisor export script sets a sentinel flag file telling the web server to return status 512, drains the results table into a text file, removes the flag file, sorts the text file, formats, and then uploads. During this time, the web service is unavailable for 30 minutes.
Instead, it could slowly drain the results table into another database to avoid blocking the web server and then continue from there. It would also provide some way of resuming in case of failure.
Edit: It could also just check for the current export json file, drain into appending the text file, but allow resuming in case of failure.