Summary
The SynthStrip nipype node in mriqc/workflows/shared.py does not declare mem_gb, so it defaults to 0.25 GB. MRIQC's memory-aware scheduler then launches multiple SynthStrip instances concurrently, each loading the full PyTorch model (~4-6 GB). This causes OOM kills on systems with ample memory (e.g. 24 GB allocated for a single participant).
Reproducer
Run MRIQC on a multi-session BIDS subject with several T1w/T2w volumes (e.g., 4x T1w + 2x T2w):
mriqc /data /out participant \
--participant-label SUB01 \
--nprocs 8 \
--mem-gb 22 \
--no-sub
The scheduler dispatches 3-4 concurrent synthstrip nodes. Each subprocess loads synthstrip.1.pt via torch.load(), consuming ~4-6 GB. Total memory spikes to 12-20 GB for brain extraction alone, triggering OOM (exit code 137):
[Node] Error on "mriqc_wf.anatMRIQC.synthstrip_wf.synthstrip"
RuntimeError: subprocess exited with code 137.
slurmstepd: error: Detected 1 oom_kill event in StepId=3.batch.
Root cause
In mriqc/workflows/shared.py, the synthstrip node only sets num_threads:
synthstrip = pe.Node(
SynthStrip(num_threads=omp_nthreads),
name='synthstrip',
num_threads=omp_nthreads,
)
Other compute-heavy nodes in the same codebase correctly declare their memory — e.g., SpatialNormalization has mem_gb=3, segmentation has mem_gb=5 — but SynthStrip was missed when the thread-limiting fix landed in #1101.
Proposed fix
Add mem_gb to the node so the scheduler properly gates concurrent instances:
synthstrip = pe.Node(
SynthStrip(num_threads=omp_nthreads),
name='synthstrip',
num_threads=omp_nthreads,
mem_gb=6,
)
Prior work
This is a continuation of #1004, which was closed by #1101. That PR correctly added thread limiting (-n flag / torch.set_num_threads()), but the memory scheduling side was not addressed.
Environment
- MRIQC 24.0.2 (via Neurodesk neurocontainer
mriqc_24.0.2_20241108)
- Slurm-managed workstation, 16 CPUs, 32 GB RAM, 24 GB allocated
- BIDS dataset: 4x T1w + 2x T2w for a single participant across multiple sessions
Summary
The SynthStrip nipype node in
mriqc/workflows/shared.pydoes not declaremem_gb, so it defaults to 0.25 GB. MRIQC's memory-aware scheduler then launches multiple SynthStrip instances concurrently, each loading the full PyTorch model (~4-6 GB). This causes OOM kills on systems with ample memory (e.g. 24 GB allocated for a single participant).Reproducer
Run MRIQC on a multi-session BIDS subject with several T1w/T2w volumes (e.g., 4x T1w + 2x T2w):
mriqc /data /out participant \ --participant-label SUB01 \ --nprocs 8 \ --mem-gb 22 \ --no-subThe scheduler dispatches 3-4 concurrent
synthstripnodes. Each subprocess loadssynthstrip.1.ptviatorch.load(), consuming ~4-6 GB. Total memory spikes to 12-20 GB for brain extraction alone, triggering OOM (exit code 137):Root cause
In
mriqc/workflows/shared.py, the synthstrip node only setsnum_threads:Other compute-heavy nodes in the same codebase correctly declare their memory — e.g.,
SpatialNormalizationhasmem_gb=3,segmentationhasmem_gb=5— but SynthStrip was missed when the thread-limiting fix landed in #1101.Proposed fix
Add
mem_gbto the node so the scheduler properly gates concurrent instances:Prior work
This is a continuation of #1004, which was closed by #1101. That PR correctly added thread limiting (
-nflag /torch.set_num_threads()), but the memory scheduling side was not addressed.Environment
mriqc_24.0.2_20241108)