Process `NFCORE_SCDOWNSTREAM:SCDOWNSTREAM:FINALIZE:ADATA_TORDS (merged)` terminated with an error exit status

[royfrancis]

Description of the bug

I got an error when running test on my HPC.

Command used and terminal output

nextflow run nf-core/scdownstream \
  -r dev \
  -profile test,singularity,uppmax \
  --project xxxxxxxxx \
  -resume \
  --outdir results \
  -with-report report.html \
  -with-timeline timeline.html \
  -with-dag dag.png

-[nf-core/scdownstream] Pipeline completed successfully, but with errored process(es) -
[1d/e035d1] NOTE: Process `NFCORE_SCDOWNSTREAM:SCDOWNSTREAM:FINALIZE:ADATA_TORDS (merged)` terminated with an error exit status (1) -- Error is ignored

The full error from that python code:

INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
INFO:    Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred
Traceback (most recent call last):
  File "/crex/proj/naiss2024-23-571/nobackup/roy/scdownstream/work/1d/e035d17b9a14f46856e3347b387c23/.command.sh", line 33, in <module>
    adata = ad.read_h5ad("merged.h5ad")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 260, in read_h5ad
    adata = read_dispatched(f, callback=callback)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/experimental/_dispatch_io.py", line 42, in read_dispatched
    return reader.read_elem(elem)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/utils.py", line 212, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 282, in read_elem
    return self.callback(read_func, elem.name, elem, iospec=iospec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 244, in callback
    k: read_dispatched(elem[k], callback)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/experimental/_dispatch_io.py", line 42, in read_dispatched
    return reader.read_elem(elem)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/utils.py", line 212, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 282, in read_elem
    return self.callback(read_func, elem.name, elem, iospec=iospec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 258, in callback
    return func(elem)
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 338, in read_mapping
    return {k: _reader.read_elem(v) for k, v in elem.items()}
               ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/utils.py", line 212, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 282, in read_elem
    return self.callback(read_func, elem.name, elem, iospec=iospec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 258, in callback
    return func(elem)
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 338, in read_mapping
    return {k: _reader.read_elem(v) for k, v in elem.items()}
               ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/utils.py", line 212, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 282, in read_elem
    return self.callback(read_func, elem.name, elem, iospec=iospec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 258, in callback
    return func(elem)
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 338, in read_mapping
    return {k: _reader.read_elem(v) for k, v in elem.items()}
               ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/utils.py", line 212, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 277, in read_elem
    read_func: Read = self.registry.get_read(
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 171, in get_read
    raise IORegistryError._from_read_parts("read", self.read, src_type, spec)
anndata._io.specs.registry.IORegistryError: No read method registered for IOSpec(encoding_type='null', encoding_version='0.1.0') from <class 'h5py._hl.dataset.Dataset'>. You may need to update your installation of anndata.
Error raised while reading key 'layer' of <class 'h5py._hl.dataset.Dataset'> from /uns/bbknn-global-0.5_characteristic_genes/params

Relevant files

.nextflow.log

System information

Nextflow: 25.10.2
nf-core: 3.5.1
Executor: slurm
Container engine: Apptainer
OS: CentOS Fedora Rocky Linux 9.7
nf-core/scdownstream: 0.0.1dev

[royfrancis]

Just realized that I am running Nextflow: 25.10.2 and the nextflow.config has nextflowVersion = '!>=25.04.0' in it. Not sure if it's related to that.

[nictru]

Hey, I am aware of this issue, and I was planning to wait with this until the bioconductor bulk release in bioconda is processed. After this is done, we can switch to using the official AnndataR package for making the conversion to RDS.

In the current version of the pipeline, errors in the ANNDATA_TORDS step should be ignored and thus should not lead to pipeline failure.