Observed several times in the past the following problem in the job-controller container, when the pod is closing:
$ kubectl get pods | grep reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7
reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7 1/2 NotReady 0 15h
$ kubectl logs reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7 job-controller | tail -25
2023-02-06 17:03:32,462 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:32,500 | werkzeug | Thread-96 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "POST /jobs HTTP/1.1" 201 -
2023-02-06 17:03:32,505 | werkzeug | Thread-97 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:32,509 | werkzeug | Thread-98 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:33,641 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:35,518 | werkzeug | Thread-99 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:35] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:38,526 | werkzeug | Thread-100 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:38] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:41,535 | werkzeug | Thread-101 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:41] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:44,543 | werkzeug | Thread-102 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:44] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:47,552 | werkzeug | Thread-103 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:47] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:50,560 | werkzeug | Thread-104 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:50] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Grabbing pod reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2-vbt6v logs ...
2023-02-06 17:03:52,951 | root | kubernetes_job_monitor | INFO | Storing job logs: ccb047dc-2df9-43c9-9a2a-4f7e9545596e
2023-02-06 17:03:52,991 | root | kubernetes_job_monitor | INFO | Cleaning Kubernetes job reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 ...
2023-02-06 17:03:53,072 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:53,072 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:53,077 | root | kubernetes_job_monitor | INFO | New Pod event received: DELETED
2023-02-06 17:03:53,077 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:53,569 | werkzeug | Thread-105 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:53] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 18:37:43,804 | root | kubernetes_job_monitor | ERROR | Error while connecting to Kubernetes API: (410)
Reason: Expired: too old resource version: 171643461 (171656967)
Let's investigate the root cause (and possibly upgrade Kubernetes cluster version?)
Observed several times in the past the following problem in the job-controller container, when the pod is closing:
Let's investigate the root cause (and possibly upgrade Kubernetes cluster version?)