How to speed up

[wangjiawen2013]

Hi,
When I ran mhcquant on our local ubuntu server

nextflow run nf-core/mhcquant \
    -profile test,singularity \
    --outdir ./mhcquant_test_results

Now the workflow ran slowly, and only one CPU and a few memory were used. How to speed up the progress by parallel computing ?

[jonasscheid]

Hi @wangjiawen2013,

Speed is heavily dependent on your setup. I imagine that running the pipeline the first time, pulling all singularity containers might take a couple of minutes. In general the test profile is very fast.

If you want to optimize for speed on your dataset, here is a quick excursus:
nextflow intrinsically parallelizes processes. Ressources for processes in nf-core pipelines are monitored in conf/base.config. The labels are specified in each module at the top (see here). These basic ressource allocations cover a broad range of use cases. E.g. Brukers TDF files need way more ressources compared to Thermo raw files in Comet search. If a process is run, these ressources are blocked on your machine by nextflow. If you want to optimize ressources, you can specify a custom.config and overwrite e.g. the process_high label and specify whatever ressource you want to give this label. You can also specify ressources on a per-module level (withName, withLabel, see Nextflow docs). Be aware that you might run into OOM issues.

If you want to go in more detail with these usage-specific questions, feel free to post them in the nf-core Slack #mhcquant channel.

[wangjiawen2013]

We cannot get access to zenodo in our country, so the pipeline was stucked and failed. Because MHCquant uses Docker/Singularity containerization, I cannot modify the Zenodo download link or copy files into the Docker image. This is also a limitation of container-based technologies like Docker and Singularity.

Is there any other workaround?

If I really want to change the Zenodo link, which file should I modify?

[jonasscheid]

I don't fully understand your zenodo issue. Why would you want to copy files into the docker image? And why would you change that?

It is still running right? MS2Rescore is more of the time intensive parts. What are your setup specs? Then we can speculate why it takes longer for you.

[wangjiawen2013]

I downloaed m2spip models manually because we cannot get access to zenodo, and set it use a customized config file, but it seems there are some issues about deeplc observed rentention time.

[wangjiawen2013]

Now I can run ms2rescore, cannot get the sumary becase the lack of deeplc rentention time.
https://files.slack.com/files-pri/TE6CZUZPH-F09UDNK276J/image.png

Sometimes I can get access to zenodo, and the workflow went successfully. But when I set the config manually, I cannot get deeplc retention time. This is the customized config:

params {
    ms2pip_model_dir = "/mnt/data/wangjw/models/ms2pip_immuno"
}
process {
    withName: 'NFCORE_MHCQUANT:MHCQUANT:RESCORE:MS2RESCORE' {
        containerOptions = "-v /mnt/data/wangjw/models/ms2pip_immuno:/mnt/ms2pip_models"
        ext.args = "--ms2pip_model_dir /mnt/ms2pip_models"
    }
}

and this is the command:

nextflow run /mnt/data/wangjw/pipelines/mhcquant -c custom_ms2rescore.config -profile docker --input ./HepG2_sample_sheet.tsv --fasta /mnt/data/wangjw/pipelines/mhcquant/test_data/UP000005640_9606.fasta --outdir /mnt/data/wangjw/tests/immuno2/mhcquant_test_results

[wangjiawen2013]

If I don't use the customized config, this error would occur:

executor >  local (18)
[-        ] NFCORE_MHCQUANT:MHCQUANT:PREPARE_SPECTRA:THERMORAWFILEPARSER     -
[-        ] NFCORE_MHCQUANT:MHCQUANT:PREPARE_SPECTRA:UNTAR                   -
[-        ] NFCORE_MHCQUANT:MHCQUANT:PREPARE_SPECTRA:UNZIP                   -
[-        ] NFCORE_MHCQUANT:MHCQUANT:PREPARE_SPECTRA:TDF2MZML                -
[-        ] NFCORE_MHCQUANT:MHCQUANT:PREPARE_SPECTRA:GUNZIP                  -
[29/3b7be6] NFCORE_MHCQUANT:MHCQUANT:OPENMS_DECOYDATABASE (UP000005640_9606) [100%] 1 of 1 ✔
[93/6b8f3e] NFCORE_MHCQUANT:MHCQUANT:PYOPENMS_CHROMATOGRAMEXTRACTOR (2)      [100%] 3 of 3 ✔
[8d/8fab3b] NFCORE_MHCQUANT:MHCQUANT:OPENMSTHIRDPARTY_COMETADAPTER (3)       [100%] 3 of 3 ✔
[83/2c9bab] NFCORE_MHCQUANT:MHCQUANT:OPENMS_PEPTIDEINDEXER (3)               [100%] 3 of 3 ✔
[38/ca0cf5] NFCORE_MHCQUANT:MHCQUANT:OPENMS_IDMASSACCURACY (3)               [100%] 3 of 3 ✔
[27/968c74] NFCORE_MHCQUANT:MHCQUANT:DATAMASH_HISTOGRAM (3)                  [100%] 3 of 3 ✔
[b2/0b735c] NFCORE_MHCQUANT:MHCQUANT:OPENMS_IDMERGER (HepG2_A)               [100%] 1 of 1 ✔
[7a/193aae] NFCORE_MHCQUANT:MHCQUANT:RESCORE:MS2RESCORE (HepG2_A)            [  0%] 0 of 1 ✘
[-        ] NFCORE_MHCQUANT:MHCQUANT:RESCORE:OPENMS_PSMFEATUREEXTRACTOR      -
[-        ] NFCORE_MHCQUANT:MHCQUANT:RESCORE:OPENMS_PERCOLATORADAPTER        -
[-        ] NFCORE_MHCQUANT:MHCQUANT:RESCORE:OPENMS_IDFILTER_Q_VALUE         -
[-        ] NFCORE_MHCQUANT:MHCQUANT:OPENMS_TEXTEXPORTER                     -
[-        ] NFCORE_MHCQUANT:MHCQUANT:SUMMARIZE_RESULTS                       -
[-        ] NFCORE_MHCQUANT:MHCQUANT:MULTIQC                                 -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/mhcquant] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MHCQUANT:MHCQUANT:RESCORE:MS2RESCORE (HepG2_A)'
Caused by:
  Process `NFCORE_MHCQUANT:MHCQUANT:RESCORE:MS2RESCORE (HepG2_A)` terminated with an error exit status (1)
Command executed:
  ms2rescore_cli.py \
      --psm_file HepG2_A.idXML \
      --spectrum_path . \
      --output_path HepG2_A_ms2rescore.idXML \
      --processes 12 \
      --ms2_tolerance 0.02 --ms2pip_model Immuno-HCD --ms2pip_model_dir null --rescoring_engine percolator --feature_generators deeplc,ms2pip
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MHCQUANT:MHCQUANT:RESCORE:MS2RESCORE":
      MS²Rescore: $(echo $(ms2rescore --version 2>&1) | grep -oP 'MS²Rescore \(v\K[^\)]+' ))
  END_VERSIONS
Command exit status:
  1
Command output:
  (empty)
Command error:
  /usr/local/lib/python3.12/site-packages/deeplc/feat_extractor.py:493: SyntaxWarning: invalid escape sequence '\['
    "\[.*?\]", "", atom_position_composition
  2025-11-21 09:39:05,750 INFO Percolator rescoring engine has been specified. Use the idXML containing rescoring features and run Percolator in a separate step.
  2025-11-21 09:39:05,750 INFO MS²Rescore config:
  2025-11-21 09:39:05,750 INFO {'$schema': './config_schema.json', 'ms2rescore': {'feature_generators': {'ms2pip': {'model': 'Immuno-HCD', 'ms2_tolerance': 0.02, 'model_dir': 'null'}, 'deeplc': {'deeplc_retrain': False, 'calibration_set_size': 0.15}}, 'rescoring_engine': {}, 'config_file': None, 'psm_file': 'HepG2_A.idXML', 'psm_file_type': 'infer', 'psm_reader_kwargs': {}, 'spectrum_path': '.', 'output_path': 'HepG2_A_ms2rescore.idXML', 'log_level': 'info', 'id_decoy_pattern': '^DECOY_', 'psm_id_pattern': None, 'spectrum_id_pattern': None, 'psm_id_rt_pattern': None, 'psm_id_im_pattern': None, 'lower_score_is_better': True, 'max_psm_rank_input': 10, 'max_psm_rank_output': 1, 'modification_mapping': {}, 'fixed_modifications': {}, 'processes': 12, 'rename_to_usi': False, 'fasta_file': None, 'write_report': False, 'ms2pip_model_dir': 'null'}}
  2025-11-21 09:39:06,227 INFO Removed 0 PSMs with rank >= 10.
  2025-11-21 09:39:06,235 INFO Found 3149 PSMs, of which 27.31% are decoys.
  2025-11-21 09:39:06,247 WARNING Non-mapped modifications found: {'Oxidation'}
  This can be ignored if they are Unimod modification labels.
  2025-11-21 09:39:06,253 INFO Found 0 identified PSMs with rank <= 1 at 0.01 FDR before rescoring.
  2025-11-21 09:39:06,257 INFO Adding MS²PIP-derived features to PSMs.
  2025-11-21 09:39:06,259 INFO Running MS²PIP for PSMs from run (1/3) `HepG2_rep1_small`...
  2025-11-21 09:39:07,444 INFO Downloading model_20210316_Immuno_HCD_B.xgboost to null/model_20210316_Immuno_HCD_B.xgboost...
  Traceback (most recent call last):
    File "/mnt/data/wangjw/pipelines/mhcquant/bin/ms2rescore_cli.py", line 175, in <module>
      sys.exit(main())
               ^^^^^^
    File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
      return self.main(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1082, in main
      rv = self.invoke(ctx)
           ^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
      return ctx.invoke(self.callback, **ctx.params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
      return __callback(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/mnt/data/wangjw/pipelines/mhcquant/bin/ms2rescore_cli.py", line 171, in main
      rescore_idxml(kwargs["psm_file"], kwargs["output_path"], config)
    File "/mnt/data/wangjw/pipelines/mhcquant/bin/ms2rescore_cli.py", line 83, in rescore_idxml
      rescore(config, psm_list)
    File "/usr/local/lib/python3.12/site-packages/ms2rescore/core.py", line 79, in rescore
      fgen.add_features(psm_list)
    File "/usr/local/lib/python3.12/site-packages/ms2rescore/feature_generators/ms2pip.py", line 194, in add_features
      ms2pip_results = correlate(
                       ^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/ms2pip/core.py", line 275, in correlate
      ms2pip_parallelized = _Parallelized(
                            ^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/ms2pip/core.py", line 544, in __init__
      validate_requested_xgb_model(
    File "/usr/local/lib/python3.12/site-packages/ms2pip/_utils/xgb_models.py", line 21, in validate_requested_xgb_model
      _download_model(model_file, xgboost_model_hashes[model_file], model_dir)
    File "/usr/local/lib/python3.12/site-packages/ms2pip/_utils/xgb_models.py", line 99, in _download_model
      urllib.request.urlretrieve(f"https://zenodo.org/records/13270668/files/{model}", filename)
    File "/usr/local/lib/python3.12/urllib/request.py", line 276, in urlretrieve
      raise ContentTooShortError(
  urllib.error.ContentTooShortError: <urlopen error retrieval incomplete: got only 226770048 out of 358573503 bytes>
Work dir:
  /mnt/data/wangjw/tests/immuno2/work/7a/193aaeae196284f073f6901509d5aa
Container:
  quay.io/biocontainers/ms2rescore:3.1.5--pyh7e72e81_0
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting
 -- Check '.nextflow.log' file for details