Slurm error trace is no longer visible

[poautran]

In GitLab by @woutdenolf on Jun 13, 2025, 17:08 GMT+2:

ewoksjob.apps.errors.EwoksJobExceptionCause: 

Traceback (most recent call last):
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/apps/errors.py", line 36, in replace_exception_for_client
    yield
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/apps/ewoks.py", line 53, in new_celery_task
    return execute(ewoks_task, *args, **kwargs)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/worker/slurm.py", line 87, in execute
    return future.result()
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_executor.py", line 197, in _worker_main
    result = execute(task, args, kwargs, future)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/concurrent/rest.py", line 194, in execute
    initialize(submit_kw)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/concurrent/rest.py", line 184, in initialize
    worker_proxy.initialize()
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_proxy_base.py", line 92, in initialize
    self._send_when_not_closed(initial_data)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_proxy_base.py", line 145, in _send_when_not_closed
    self._send(data)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_proxy_remote.py", line 64, in _send
    self._connection.send_data(data)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_connection_base.py", line 87, in send_data
    self._send_bytes_with_check(bheader + bdata)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_connection_base.py", line 146, in _send_bytes_with_check
    self._send_bytes(data)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_connection_tcp.py", line 88, in _send_bytes
    self._wait_client()
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/client/job_io/local/_connection_tcp.py", line 84, in _wait_client
    self._raise_on_status_error()
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/pyslurmutils/concurrent/rest.py", line 241, in raise_on_status_error
    raise exc_type(error_message) from None
pyslurmutils.client.errors.RemoteExit: SLURM job 24608175 FAILED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/apps/ewoks.py", line 27, in new_celery_task
    return celery_task(self, *args, **kwargs)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/apps/ewoks.py", line 42, in new_celery_task
    with replace_exception_for_client():
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/users/blissadm/conda/miniconda/envs/ewoksworker/lib/python3.10/site-packages/ewoksjob/apps/errors.py", line 42, in replace_exception_for_client
    raise EwoksJobException(str(e)) from cause
ewoksjob.apps.errors.EwoksJobException: SLURM job 24608175 FAILED

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoksjob/-/issues/63

[poautran]

In GitLab by @woutdenolf on Jun 13, 2025, 17:15 GMT+2:

Try to reproduce with an slurm environment that does not have ewoksjob[worker] installed.

[poautran]

In GitLab by @woutdenolf on Jun 14, 2025, 11:35 GMT+2:

This is because the python command itself on slurm failed.

To be checked whether we can show a trace of this or not.

Python exception are still visible as normal.