think about ways of making system more robust, e.g., Nuke

[rcoreilly]

• Nuke command is particularly problematic -- maybe write a list of jobs that were nuked to local client jobs.nuked file, with timestamp, and then prevent any further actions on any job in that list. That would prevent the problem of doing another command on a nuked job preventing the nuking of the directory.

• one issue is that the grunt.py command doesn't stay running, so all memory has to be via files that get written and loaded each time. Grunti however does not have such a limitation, so it could internally catch such things, though it is nicer if all the logic is in the bare grunt tool.

• A more general solution would be to keep a log of all actions taken (expiring anything more than an hour old or something) and looking through that to catch any attempts to write to nuked, deleted, or archived jobs. Maybe could also use this to actively confirm things, so e.g., grunti could try to confirm whether all nuked have been nuked and keep pulling until this comes back true. That would be good for various basic commands.

• might be useful for submit issue too -- when you try to submit something new, it waits to validate that previous jobs have been submitted (not 100% sure about what that means tho).

[rohrlich]

Putting any of the logic into grunti is not good as you point out. I like to submit from the command line but view results from grunti. Would it be possible to spawn a job that polls for "not found" for the nuked directory and blocks further actions until then?

[rcoreilly]

you mean from grunt.py python? that seems potentially brittle and difficult to manage apps that spin off -- if something happens with dir that prevents it from deleting, job never terminates, etc...