Skip to content

SynthStrip node missing mem_gb causes OOM with concurrent anatomical runs #1437

Description

@m9h

Summary

The SynthStrip nipype node in mriqc/workflows/shared.py does not declare mem_gb, so it defaults to 0.25 GB. MRIQC's memory-aware scheduler then launches multiple SynthStrip instances concurrently, each loading the full PyTorch model (~4-6 GB). This causes OOM kills on systems with ample memory (e.g. 24 GB allocated for a single participant).

Reproducer

Run MRIQC on a multi-session BIDS subject with several T1w/T2w volumes (e.g., 4x T1w + 2x T2w):

mriqc /data /out participant \
    --participant-label SUB01 \
    --nprocs 8 \
    --mem-gb 22 \
    --no-sub

The scheduler dispatches 3-4 concurrent synthstrip nodes. Each subprocess loads synthstrip.1.pt via torch.load(), consuming ~4-6 GB. Total memory spikes to 12-20 GB for brain extraction alone, triggering OOM (exit code 137):

[Node] Error on "mriqc_wf.anatMRIQC.synthstrip_wf.synthstrip"
RuntimeError: subprocess exited with code 137.
slurmstepd: error: Detected 1 oom_kill event in StepId=3.batch.

Root cause

In mriqc/workflows/shared.py, the synthstrip node only sets num_threads:

synthstrip = pe.Node(
    SynthStrip(num_threads=omp_nthreads),
    name='synthstrip',
    num_threads=omp_nthreads,
)

Other compute-heavy nodes in the same codebase correctly declare their memory — e.g., SpatialNormalization has mem_gb=3, segmentation has mem_gb=5 — but SynthStrip was missed when the thread-limiting fix landed in #1101.

Proposed fix

Add mem_gb to the node so the scheduler properly gates concurrent instances:

synthstrip = pe.Node(
    SynthStrip(num_threads=omp_nthreads),
    name='synthstrip',
    num_threads=omp_nthreads,
    mem_gb=6,
)

Prior work

This is a continuation of #1004, which was closed by #1101. That PR correctly added thread limiting (-n flag / torch.set_num_threads()), but the memory scheduling side was not addressed.

Environment

  • MRIQC 24.0.2 (via Neurodesk neurocontainer mriqc_24.0.2_20241108)
  • Slurm-managed workstation, 16 CPUs, 32 GB RAM, 24 GB allocated
  • BIDS dataset: 4x T1w + 2x T2w for a single participant across multiple sessions

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions