Retrieve job logs for submission-failure after polling

[dpmatthews]

Problem

If job submission fails, no remote job log retrieval is attempted. This makes sense if the failure happens at submission time (i.e. the job was previously in the waiting state). However, if the job was previously in the submitted state (i.e. Cylc has polled the submitted task and found it no longer in the queue) then it would make sense to attempt remote job log retrieval. As a minimum there should be a job.status file. There may also be job.out and job.err files if the job ran but failed (for some reason) to run the cylc message command to record the job as started. Not retrieving these files means we are missing useful information which might help explain the cause of the submission failure. At the moment you have to know to look on the remote platform for any log files in these cases.

Proposed Solution

Whenever there is a submission failure, run the remote job log retrieval if the task was previously in the submitted state. Note that any retrieve job logs retry delays should be ignored in this case - just attempt the log retrieval once.

[oliver-sanders]

As a minimum there should be a job.status file. There may also be job.out and job.err files if the job ran but failed

The scenario that this would make a difference in is the one in which the submit-failed state is incorrect. In this is the case, rather than running job log retrieval, shouldn't we poll the task?

If the poll finds any outputs in the status file, the status will change to succeeded/failed and job log retrieval logic will kick in as normal.

This may be a regression of historical polling changes, we used to poll tasks more aggressively to confirm state changes.

[dpmatthews]

The scenario that this would make a difference in is the one in which the submit-failed state is incorrect

That's a different scenario. Here I'm describing a case where the job ran but didn't even get as far as running the first cylc message to record the job as started. In this case, the submit-failed state was correct but there was still useful evidence present in job.out / job.err.