Skip to content

stdout and stderr are swapped by .command.run in SLURM #7299

Description

@tcezard

Bug report

When nextflow execute a process in SLURM the .command.run redirects the stdout and stderr to .command.out and .command.err respectively. But while doing so it also swaps the two streams which mean they are not sent correctly to any downstream capture.

Expected behavior and actual behavior

.command.run should execute the .command.sh capture copies of the output and error streams and leave them in their assigned stream.

Steps to reproduce the problem

params.outdir = "logs"
def outdir = file(params.outdir).toAbsolutePath().normalize()

workflow {
    // clusterOptions writes directly to this path when the job starts,
    // so it must exist before the job is submitted.
    outdir.mkdirs()

    ECHO_STREAMS()
}

process ECHO_STREAMS {
    clusterOptions {
        [
            "--output=${outdir}/stdout.log".toString(),
            "--error=${outdir}/stderr.log".toString()
        ]
    }

    script:
    """
    echo "STDOUT_MESSAGE: this line was written to stdout"
    echo "STDERR_MESSAGE: this line was written to stderr" 1>&2
    """
}
NXF_SYNTAX_PARSER=v1  nextflow run reproduce_nextflow_stdout_stderr_swap.nf --outdir log
cat log/stdout.log
cat log/stderr.log

Program output

Note the order of the two message compared to the order of the two command above.

STDERR_MESSAGE: this line was written to stderr
STDOUT_MESSAGE: this line was written to stdout

Environment

  • Nextflow version: 26.04.4
  • Java version: 21.0.3
  • Operating system: Linux
  • Bash version: GNU bash, version 5.1.8

Additional context

I ran this on SLURM to capture the output of .command.run but this probably applies elsewhere.
I believe that the culprit is this line in the .command.run:

(set -o pipefail; (nxf_launch | tee .command.out) 3>&1 1>&2 2>&3 | tee .command.err)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions