Problem using workflow.recurse with multiple parallel states: process reused error

[xuyinnnan]

Hi Nextflow community,

I’m trying to use the workflow.recurse preview feature to implement an iterative workflow, but I’m running into issues when multiple mod_name values are processed in parallel.

Background

I have multiple (sample_id, mod_name) combinations, and each one needs to independently run an iterative workflow (ITERATION_LOOP).
Because recurse only accepts a single value, I wrapped it inside a flatMap so that each state runs its own recursion:

iteration_state = first_loop_input
    .flatMap { state_list ->
        def ...
        println "=========================================="
        println "[${sample_id}][${mod_name}] start"
        println "=========================================="

        ITERATION_LOOP
            .recurse(initial_state)
            .until { state ->
                if (!state || state.size() == 0) {
                    println "[WARN] empty"
                    return true
                }

                def ...
                if (stop) {
                    println "[${sample_id}][${mod_name}] stop: iteration=${iteration}, bool=${bool_val}"
                }
                return stop
            }
            .last()
    }

This avoids the “single value only” limitation of recurse, and logically each (sample_id, mod_name) should have its own independent iteration.

Problem

When running this, I get the following error:

ERROR ~ Invalid method invocation `doCall` with arguments: 
DataflowVariable(value=null) (groovyx.gpars.dataflow.DataflowVariable) 
on _closure113 type

Process 'STEP_CV_FILTER' has been already used -- 
If you need to reuse the same component, include it with a different name 
or include it in a different workflow context

The error points to:
workflows/../modules/cluster_realignment/iteration_loop.nf:22

My understanding
recurse itself works fine for a single iterative chain
When multiple mod_name values enter the recursion in parallel:
The workflow ITERATION_LOOP is invoked multiple times
Processes inside it (e.g. STEP_CV_FILTER) appear to be reused across recursion contexts
This causes Nextflow to complain that the process has “already been used”

Questions:
Is workflow.recurse designed to support multiple parallel recursion contexts?
Is it invalid to call the same workflow with recurse multiple times inside flatMap?
What is the recommended pattern for:
Running independent iterative loops
One loop per (sample_id, mod_name)

Any guidance or examples would be greatly appreciated.
Thanks in advance!

[bentsherman]

Hi @xuyinnnan , the basic problem is that you can't call processes and workflows inside an operator closure (like flatMap). It's because Nextflow needs to know the full process DAG before executing the pipeline.

In practice this will limit the number of "levels" of iteration / recursion. Sometimes you can refactor the workflow logic so that you call the iterative workflow once on a single merged channel, but it might not be possible depending on what you are doing. I would need to see a complete example to be sure.

In such cases, you might have to refactor the iterative workflow as a process so that the recursion is handled by the process script. You can often recover the parallelism you would have had from a workflow by using more cpus and using bash parallelism.

[xuyinnnan]

Alright, I understand the issue now. Thank you. I'll adjust the process to implement this.