K8SPXC-1686: delete job and its pods if runningDeadlineSeconds has been exceeded

[mayankshah1607]

CHANGE DESCRIPTION

Problem:

Backup job pods can remain in the cluster after runningDeadlineSeconds has exceeded. There is a chance that eventually the pods start running and a backup takes place (which would be unexpected since we marked the backup has failed)

Cause:

We leave the job behind after marking the backup as failed

Solution:
Delete the job after backup has failed due to runningDeadlineSeconds exceeded.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PXC version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
auto-tuning-8-0	passed	00:18:39
allocator-8-0	passed	00:11:50
backup-storage-tls-8-0	passed	00:21:20
cross-site-8-0	passed	00:33:04
custom-users-8-0	passed	00:11:14
demand-backup-cloud-8-0	passed	00:55:39
demand-backup-encrypted-with-tls-8-0	passed	00:44:07
demand-backup-8-0	passed	00:42:13
demand-backup-flow-control-8-0	passed	00:10:22
demand-backup-parallel-8-0	passed	00:08:50
demand-backup-without-passwords-8-0	passed	00:15:23
haproxy-5-7	passed	00:14:58
haproxy-8-0	passed	00:14:21
init-deploy-5-7	passed	00:17:00
init-deploy-8-0	passed	00:16:52
limits-8-0	passed	00:12:17
monitoring-2-0-8-0	passed	00:22:18
monitoring-pmm3-8-0	passed	00:22:33
one-pod-5-7	passed	00:14:30
one-pod-8-0	passed	00:14:41
pitr-8-0	passed	00:42:03
pitr-gap-errors-8-0	passed	00:56:00
proxy-protocol-8-0	passed	00:09:39
proxy-switch-8-0	passed	00:13:26
proxysql-sidecar-res-limits-8-0	passed	00:08:07
proxysql-scheduler-8-0	passed	00:17:16
pvc-resize-5-7	passed	00:15:58
pvc-resize-8-0	passed	00:14:21
recreate-8-0	passed	00:17:15
restore-to-encrypted-cluster-8-0	passed	00:26:11
scaling-proxysql-8-0	passed	00:08:27
scaling-8-0	passed	00:10:53
scheduled-backup-5-7	passed	01:05:01
scheduled-backup-8-0	passed	01:01:51
security-context-8-0	passed	00:26:07
smart-update1-8-0	passed	00:30:56
smart-update2-8-0	passed	00:37:46
storage-8-0	passed	00:10:53
tls-issue-cert-manager-ref-8-0	passed	00:08:58
tls-issue-cert-manager-8-0	passed	00:10:01
tls-issue-self-8-0	passed	00:13:22
upgrade-consistency-8-0	passed	00:11:33
upgrade-haproxy-5-7	passed	00:23:55
upgrade-haproxy-8-0	passed	00:24:47
upgrade-proxysql-5-7	passed	00:13:18
upgrade-proxysql-8-0	passed	00:15:06
users-5-7	passed	00:24:15
users-8-0	passed	00:26:13
validation-hook-8-0	passed	00:01:49

Summary	Value
Tests Run	49/49
Job Duration	03:00:53
Total Test Time	17:47:57

commit: 21635d9
image: perconalab/percona-xtradb-cluster-operator:PR-2270-21635d9b