Alarm for workers

[ahellander]

Should we turn off workers automatically after idle for some time? To save on resources.

[briandrawert]

Proposal: Each WorkerGroup has a setup option: "auto-shutoff time" (in hours). We use a timer like we do with the StochSS workers, and integrate with molnsutil to restart the timer each time a job is run on a worker node. Issue: If you use ipython.parallel directly, nodes will just shutdown unexpectedly on you. Partial solution: change ipython.parallel profile from 'default' to 'molnsutil' to make it more obvious that is what is it for.

[ahellander]

Spontaneously this seems reasonable, but I do worry about the ipython parallel issue. Maybe we could have a timer that is based on cpu load?