Cancel doesn't always work - issue is slurmId

[rohrlich]

I believe the failure to cancel is due to the wrong SlurmId and it always seems to be one less than it really is (e.g. the really ID is 268743 but grunt has it as 268742. This may only be a problem with array jobs. Anyone else see this occasional problem?

Workaround is of course to execute scancel on the server with the correct ID

[rcoreilly]

The array job numbering may be off, however it has been working for me just in last day. Not sure what might be different? Are you using the newer array grunter script as found in grunt/grunter_array.py? there was an older version that did things a bit differently.

[rohrlich]

Yes the latest version. Only happens sometimes but I don't have a lot of data because I don't often cancel. I can start watching to see if the ID is correct and update the ticket at some point.

[rohrlich]

slurmId definitely off by one. I have the latest grunt code. Is there anything in the grunter code that might be out of date and I should change?

[rohrlich]

Here is some likely helpful information --
job.slurmid has 3 values, e.g.
272125 (this is the one that shows up in the grunt slurm id column)
272126 (actual jobs)
272127 (waiting around to cleanup - I believe)

Here is what squeue shows
272127 high ac1_roh0 rohrlich PD 0:00 1 (Dependency)
272126_0 high ac1_roh0 rohrlich R 2:34 1 agate-12
272126_1 high ac1_roh0 rohrlich R 2:34 1 agate-12
272126_2 high ac1_roh0 rohrlich R 2:34 1 agate-12
272126_3 high ac1_roh0 rohrlich R 2:34 1 agate-16
272126_4 high ac1_roh0 rohrlich R 2:34 1 agate-16