Slurm runs can get so big that we can't list all the job IDs on the sacct command line and crash

[adamnovak]

This was reported in ComparativeGenomicsToolkit/cactus#1465

We attempt to put too many job IDs into an sacct command, and we get an argument list too long error. That's not among the ones that prompt us to fall back to scontrol here:

toil/src/toil/batchSystems/slurm.py

Line 301 in 5a8090e

try:

So the net result is that Toil crashes.

We should set up to run multiple sacct commands when we have too many jobs for one, and if weird stuff goes wrong with launching sacct in general (not just if we launch it and it fails) we should fall back to scontrol.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1632