After removing failures there are no samples left

[kittychaseNC]

Description of the bug

Pipeline fails, no samples left but phoenix QC passed all samples

.nextflow.log

outdir_GRiPHin_Summary.xlsx

Command used and terminal output

nextflow run cdcgov/phylophoenix -profile docker --outdir /mnt/WGSData/MolecularEpi/250728/outdir_phylo --indir /mnt/WGSData/MolecularEpi/250728/outdir/

ERROR ~ Error executing process > 'PHYLOPHOENIX_WF:PHYLOPHOENIX:GRIPHIN_WORKFLOW:GRIPHIN (1)'

Caused by:
  Process `PHYLOPHOENIX_WF:PHYLOPHOENIX:GRIPHIN_WORKFLOW:GRIPHIN (1)` terminated with an error exit status (1)


Command executed:

  # rename input samplesheet so it gets passed along.
  if [[ samplesheet_test_illumina_amplicon.csv != "Directory_samplesheet.csv" ]]; then
      mv samplesheet_test_illumina_amplicon.csv Directory_samplesheet.csv
  fi

  GRiPHin.py --ar_db ResGANNCBI_20230517_srst2.fasta --coverage 30 --samplesheet Directory_samplesheet.csv  --phoenix

  cat <<-END_VERSIONS > versions.yml
  "PHYLOPHOENIX_WF:PHYLOPHOENIX:GRIPHIN_WORKFLOW:GRIPHIN":
     python: $(python --version | sed 's/Python //g')
     phoenix_base_container: base_v2.1.0
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/home/EADS/kchase4/.nextflow/assets/cdcgov/phylophoenix/bin/GRiPHin.py", line 1565, in <module>
      main()
    File "/home/EADS/kchase4/.nextflow/assets/cdcgov/phylophoenix/bin/GRiPHin.py", line 1534, in main
      QC_reason, MLST_scheme_1, MLST_scheme_2, MLST_type_1, MLST_type_2, MLST_alleles_1, MLST_alleles_2, MLST_source_1, MLST_source_2 = Get_Metrics(args.phoenix, args.scaffolds, args.set_coverage, srst2_ar_df, pf_df, ar_df, ats, kraken_trim, kraken_trim_report, kraken_wtasmbld_report, kraken_wtasmbld, quast_report, busco_short_summary, asmbld_ratio, gc, sample_name, mlst_file, fairy_file, gamma_ar_file, gamma_pf_file, gamma_hv_file, fast_ani_fi ar_dic)
    File "/home/EADS/kchase4/.nextflow/assets/cdcgov/phylophoenix/bin/GRiPHin.py", line 743, in Get_Metrics
      Q30_R1_per, Q30_R2_per, Total_Raw_Seq_bp, Total_Raw_reads, Total_Trimmed_bp, Paired_Trimmed_reads, Total_Trimmed_reads, Trim_Q30_R1_percent, Trim_Q30_R2_percent = get_Q30(trim_stats, raw_stats)
    File "/home/EADS/kchase4/.nextflow/assets/cdcgov/phylophoenix/bin/GRiPHin.py", line 79, in get_Q30
      data_df = pd.read_csv(trim_stats, sep='\t', header=0)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
      return func(*args, **kwargs)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
      return _read(filepath_or_buffer, kwds)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 482, in _read
      parser = TextFileReader(filepath_or_buffer, **kwds)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
      self._engine = self._make_engine(self.engine)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
      return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in __init__
      self._open_handles(src, kwds)
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/parsers/base_parser.py", line 229, in _open_handles
      errors=kwds.get("encoding_errors", "strict"),
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/common.py", line 614, in get_handle
      storage_options=storage_options,
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/common.py", line 312, in _get_filepath_or_buffer
      with urlopen(req_info) as req:
    File "/pyenv/versions/3.7.12/lib/python3.7/site-packages/pandas/io/common.py", line 212, in urlopen
      return urllib.request.urlopen(*args, **kwargs)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 222, in urlopen
      return opener.open(url, data, timeout)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 531, in open
      response = meth(req, response)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 641, in http_response
      'http', request, response, code, msg, hdrs)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 569, in error
      return self._call_chain(*args)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 503, in _call_chain
      result = func(*args)
    File "/pyenv/versions/3.7.12/lib/python3.7/urllib/request.py", line 649, in http_error_default
      raise HTTPError(req.full_url, code, msg, hdrs, fp)
  urllib.error.HTTPError: HTTP Error 404: Not Found

Work dir:
  /mnt/WGSData/work/d7/e42ff1d4420e6c78ea7f9e6dd064c9

Container:
  quay.io/jvhagey/phoenix:base_v2.1.0

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Relevant files

No response

System information

No response

[jvhagey]

There error posted is different from what is in the .nextflow.log so I'm wondering which is the actual problem? I was able to pull the samples from NCBI and run them and can't replicate the error. This are the commands I used:

nextflow run cdcgov/phoenix -r v2.1.1 -latest -profile singularity,cdcsge -entry SRA --input_sra sra_samplesheet.csv --kraken2db $KRAKEN_DB_v2_2 --outdir phx_output_v2.1.1

and then:

nextflow run cdcgov/phylophoenix -latest -profile singularity,cdcsge --outdir NC_outdir_phylo --indir phx_output/

Kraken2 db: Standard-8 built on 4/2/2025

This directory has a completed phx run correct, /mnt/WGSData/MolecularEpi/250728/outdir/ ? What kind of system are you running this on, is that different from how you ran phx? I can send you the results via email if you would like.

[kittychaseNC]

Hi Jill, I may have attached the wrong log file my apologies but the error we are seeing: After removing failures no samples left. There is a completed successful run in the folder mnt/WGSData/MolecularEpi/250728/outdir/and all of those samples passed QC. We are running this on our line server, -profile docker. In the interim, yes could you please send me the phylophoenix output files for those samples via email. Kitty Chase Sequencing and Bioinformatics Response Unit SaBR

…

________________________________ From: Jill V. Hagey, MS, PhD ***@***.***> Sent: Wednesday, July 30, 2025 14:13 To: CDCgov/phylophoenix ***@***.***> Cc: Chase, Kitty ***@***.***>; Author ***@***.***> Subject: [External] Re: [CDCgov/phylophoenix] After removing failures there are no samples left (Issue #4) CAUTION: External email. Do not click links or open attachments unless verified. Report suspicious emails with the Report Message button located on your Outlook menu bar on the Home tab. [https://avatars.githubusercontent.com/u/21023514?s=20&v=4]jvhagey left a comment (CDCgov/phylophoenix#4)<#4 (comment)> There error posted is different from what is in the .nextflow.log so I'm wondering which is the actual problem? I was able to pull the samples from NCBI and run them and can't replicate the error. This are the commands I used: nextflow run cdcgov/phoenix -r v2.1.1 -latest -profile singularity,cdcsge -entry SRA --input_sra sra_samplesheet.csv --kraken2db $KRAKEN_DB_v2_2 --outdir phx_output_v2.1.1 and then: nextflow run cdcgov/phylophoenix -latest -profile singularity,cdcsge --outdir NC_outdir_phylo --indir phx_output/ Kraken2 db: Standard-8 built on 4/2/2025 This directory has a completed phx run correct, /mnt/WGSData/MolecularEpi/250728/outdir/ ? What kind of system are you running this on, is that different from how you ran phx? I can send you the results via email if you would like. — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS2PY5ILXGNNE25GPPK7CFL3LEDLFAVCNFSM6AAAAACCRCY6L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMZXGM3DSNBUHA>. You are receiving this because you authored the thread.

________________________________ Email correspondence to and from this address is subject to the North Carolina Public Records Law and may be disclosed to third parties by an authorized State official. Unauthorized disclosure of juvenile, health, legally privileged, or otherwise confidential information, including confidential information relating to an ongoing State procurement effort, is prohibited by law. If you have received this email in error, please notify the sender immediately and delete all records of this email.

[jvhagey]

A couple of follow up questions, is this the first time you used the pipeline? I was wondering if it ever worked for you and just failed with this set of samples or if its never worked? Also, can you try using --force ? which "should" ignore that step.

[kittychaseNC]

This is the first time using the pipeline. We will try -force and see what happens. Kitty Chase Saving the world one in silico analysis at a time Lead Bioinformatics Scientist Sequencing and Bioinformatics Response Unit SaBR 919-807-8835 office ***@***.******@***.***> North Carolina State Laboratory of Public Health 4312 District Dr 1918 Mail Service Center Raleigh, NC 27699-1918 From: Jill V. Hagey, MS, PhD ***@***.***> Sent: Thursday, July 31, 2025 2:30 PM To: CDCgov/phylophoenix ***@***.***> Cc: Chase, Kitty ***@***.***>; Author ***@***.***> Subject: [External] Re: [CDCgov/phylophoenix] After removing failures there are no samples left (Issue #4) CAUTION: External email. Do not click links or open attachments unless verified. Report suspicious emails with the Report Message button located on your Outlook menu bar on the Home tab. [https://avatars.githubusercontent.com/u/21023514?s=20&v=4]jvhagey left a comment (CDCgov/phylophoenix#4)<#4 (comment)> A couple of follow up questions, is this the first time you used the pipeline? I was wondering if it ever worked for you and just failed with this set of samples or if its never worked? Also, can you try using --force ? which "should" ignore that step. - Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS2PY5KPVTS7UO6PCVS6W6D3LJOBVAVCNFSM6AAAAACCRCY6L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNBQHE2TGNZXGA>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

…

________________________________ Email correspondence to and from this address is subject to the North Carolina Public Records Law and may be disclosed to third parties by an authorized State official. Unauthorized disclosure of juvenile, health, legally privileged, or otherwise confidential information, including confidential information relating to an ongoing State procurement effort, is prohibited by law. If you have received this email in error, please notify the sender immediately and delete all records of this email.

[kittychaseNC]

tried --force. Same error:

.nextflow.log

ERROR ~ Error executing process > 'PHYLOPHOENIX_WF:PHYLOPHOENIX:GRIPHIN_WORKFLOW:REMOVE_FAILURES (GRiPHin_Summary.tsv)'

Caused by:
Process PHYLOPHOENIX_WF:PHYLOPHOENIX:GRIPHIN_WORKFLOW:REMOVE_FAILURES (GRiPHin_Summary.tsv) terminated with an error exit status (1)

[wchen190]

Tried adding -latest to the command, got the same error.

Pulling cdcgov/phylophoenix ...
Already-up-to-date

Please test it with docker, thanks.

[jvhagey]

Hi, I know its been a very long time (sorry). We have made some upgrades to the dev branch can you give that a try when you get a chance? Also, @aginnimb found another bug that was being caused by file names that were getting truncated so we should be putting a fix in for that soon. Just wanted to update ya'll that we are working to get this fixed still.

[wch190191]

Just tried again and got the same error as before, but looks like v1.1.0 is not found, and v1.0dev-g5c70948 is the one I used.

[hkunerth]

Quick question: I'm trying to run PhyloPHoeNIx on some samples that have failed PHoeNIx. Am I correct that the main branch does not have the --force option implemented (other than throwing the warning message) and that I will need to pull the dev branch for it to work?