PBS: use `qstat -f` when polling in order to extract comment for stuck submitted jobs?

[MetRonnie]

Problem

Sometimes a PBS job can become stuck in the submitted state (e.g. due to a problem with a queue). It may not be possible to tell the problem from Cylc, without manually running qstat -f

E.g.

comment = Not Running: Placement set is too small: Not enough free nodes available

Proposed Solution

Adjust the PBS job runner manager to use qstat -f -F json as the poll command, and parse the results.

The comment could be included in the poll output so would show in the job activity log, or could be sent by cylc message so it would show in the UI.

[oliver-sanders]

This is really a problem for the batch system to solve rather than Cylc. These jobs will remain in the queue (even if we train Cylc to ignore them) and need to be handled/purged.

In this particular example, PBS cannot schedule the job as there isn't sufficient resource in the HPC to fulfil the requirements. So rather than fail the submission, PBS keeps it in the queue in the hilarious hope that one day someone will equip the HPC with nodes which would be able to fulfil the request (Another society will arise, one day there will be lemon soaked paper napkins).

How much of this (and the comment format) is down to the PBS implementation used and configuration applied I don't know.

[MetRonnie]

In the example above, the HPC quad that was submitted to had been taken offline temporarily. So in theory it would have started running when the quad came back. However it would indeed have been more useful for the job to submit-fail so the operators could have re-triggered on a live quad.

[MetRonnie]

I think there may be cases where the job is waiting for resource (for a long time due to contention), it might still be useful for a poll to return the comment?

[oliver-sanders]

It's not "trivially" easy to get Cylc to ingest comments from PBS. We poll multiple jobs in a single qstat command and expect a table-like output with one line per job.

qstat -f would return many lines per job with the potential to saturate stdout, so probably not a runner. But, from the PBS Pro user guide it looks like we can get the comment using the -s option. This keeps the table format, inserting a second line for each job to hold the comment (two lines per job). Would still need to do some bodging to teach Cylc how to interpret these results, but should be possible. But of course, that's providing that the comment field and qstat -s output are consistent between different PBS implementations 🤦.

Having retrieved the comment, what would we actually do with it? I don't think the comment would end up in the job-activity.log file, event if it were present in the poll output (we dump the results of poll output processing, not the raw output itself). And it won't get logged in the scheduler either (possibly with the exception of debug mode? or is that debug mode but only if the poll command fails?).

If we do manage to get the message into the scheduler log or job-activity.log, what value would it have / purpose would it serve? The suggestion seems to be that the comment isn't sufficient to diagnose any issue - which is rather unhelpful. Also, the messages produced by PBS Pro can be somewhat misleading from a user perspective. E.g, "cannot run" might simply mean, this iteration of the PBS scheduling-loop couldn't find a slot to fit the job into, but a user might interpret this as a more terminal message which might be confusing.

[MetRonnie]

In terms of parsing the result, there is the -F json option as mentioned above, which seems to have been around since at least the 2019 version (earliest I found in a quick search). Not sure where the limit on saturating stdout comes from?

If we do manage to get the message into the scheduler log or job-activity.log, what value would it have / purpose would it serve?

The same purpose that operators have to currently go to qstat for to find out why a job is in the submitted state for a long time.

[MetRonnie]

Removed question label as we have agreed in-person at the Met Office this would be useful. However due to fragility of parsing non-JSON polling output, we should go for qstat -f -F json and this will require some rewriting of the job runner manager / polling code (would be a step towards tackling #4678)