Error: index file too big for skippable zstd frame Error: could not append S1R index

[Nixon-hk]

Dear Development Team,

I encountered an error message when I try to do imputation:

Error: index file too big for skippable zstd frame
Error: could not append S1R index

, and I saw the disk usage running out, so I tried to impute by splitting chromosome into 5 Mb chunks, but I still encountered the error message again.

As my reference panel and the phased file are just 200MB and 1.2GB, I do not understand why I encountered such memory problems, as my server can handle well with large files.

Do you know what are the possible causes of this problem ? Many Thanks. Have a nice day.

Yours sincerely,

Nixon So

[jonathonl]

That error message is indicative of a disk write error. This could be from running out of space on the disk pointed to by --output or the default temp directory. See --temp-prefix for specifying a custom temp space. This error message could also be caused by other write errors, such as lacking write permissions in either the output or temp directory.

[Nixon-hk]

That error message is indicative of a disk write error. This could be from running out of space on the disk pointed to by --output or the default temp directory. See --temp-prefix for specifying a custom temp space. This error message could also be caused by other write errors, such as lacking write permissions in either the output or temp directory.

Dear Johnathon,

Thanks for your prompt reply, I examined the disk space is very large enough to accommodate these output files, and I also tried to split the chromosome into chunks to run imputation:

Here is my shell script:

#!/bin/bash

CHR=22
CHRLENGTH=50818468
CHUNK_SIZE=2000000

REFERENCE="chr_22_reference.msav"
TARGET_VCF="phased_chr22.vcf.gz"

for ((START=1; START<=CHRLENGTH; START+=CHUNK_SIZE))
do
END=$((START + CHUNK_SIZE - 1))
if [ $END -gt $CHRLENGTH ]; then
END=$CHRLENGTH
fi

echo "current processing region: $CHR:$START-$END"

CHUNK_VCF="phased_chr22_${START}_${END}.vcf.gz"
bcftools view -r ${CHR}:${START}-${END} $TARGET_VCF -Oz -o $CHUNK_VCF
bcftools index $CHUNK_VCF

OUTPUT="chr22_imputed_${START}_${END}.vcf.gz"
minimac4 --temp-prefix /media/saadmin/Data8/minimac4_temp --threads 20 $REFERENCE $CHUNK_VCF -o $OUTPUT -r ${CHR}:${START}-${END}

rm $CHUNK_VCF $CHUNK_VCF.csi
done

Please take a look. Many Thanks. Have a nice day.

Yours sincerely,

Nixon So

[jonathonl]

If this is an accurate copy and paste of your shell script, then there are a few errors in your script that will cause unexpected outcomes:

1. The argument you are passing to -o needs a $ before OUTPUT.
2. The -r is using a "minus" symbol instead of a hyphen.
3. The CHR component of the -r argument is also missing a $ before the variable name.

You need to also ensure that you have write permissions to file directories that you are specifying as output and temp. You also need to ensure the full paths to the parent directories of the output and temp paths exist.

[Nixon-hk]

If this is an accurate copy and paste of your shell script, then there are a few errors in your script that will cause unexpected outcomes:

1. The argument you are passing to -o needs a $ before OUTPUT.
2. The -r is using a "minus" symbol instead of a hyphen.
3. The CHR component of the -r argument is also missing a $ before the variable name.

You need to also ensure that you have write permissions to file directories that you are specifying as output and temp. You also need to ensure the full paths to the parent directories of the output and temp paths exist.

Dear Johnathan,

Thanks for your reply, for your examined shell script, it should be some error during the copy and paste from my terminal, here is the complete script:

#!/bin/bash

CHR=22
CHRLENGTH=50818468 # total length of chr22
CHUNK_SIZE=5000000 # 5 Mb chunks

REFERENCE="chr_22_reference.msav"
TARGET_VCF="phased_chr22.vcf.gz"

for ((START=1; START<=CHRLENGTH; START+=CHUNK_SIZE))
do
END=$((START + CHUNK_SIZE - 1))
if [ $END -gt $CHRLENGTH ]; then
END=$CHRLENGTH
fi

echo "Processing region: $CHR:$START-$END"

CHUNK_VCF="phased_chr22_${START}_${END}.vcf.gz"
bcftools view -r ${CHR}:${START}-${END} $TARGET_VCF -Oz -o $CHUNK_VCF
bcftools index $CHUNK_VCF

OUTPUT="chr22_imputed_${START}_${END}.vcf.gz"
minimac4 --temp-prefix /media/saadmin/Data8/minimac4_temp --threads 20 $REFERENCE $CHUNK_VCF -o $OUTPUT -r ${CHR}:${START}-${END}

rm $CHUNK_VCF $CHUNK_VCF.csi

and no matter I tried smaller region of each chunk, the program always struck at this point:

Completed 101400 of 488377 samples
Completed 101600 of 488377 samples
Completed 101800 of 488377 samples
Error: index file too big for skippable zstd frame
Error: could not append S1R index

And here is the screenshot of the time when the programme crash:

and I already set the directory which has write permissions as well before running the shell script.

Please take a look. Many Thanks. Have a nice day.

Yours sincerely,

Nixon So

[jonathonl]

This does look like a disk space issue. Note that disk requirements are different than RAM requirements. Imputing a half million samples will result in a very large output file. The exact size depends on the reference panel you are using. I would subset the samples instead of the region. I would create a VCF including only the first 1,000 samples and impute that. You can then extrapolate the total disk space needed by multiplying the output file size for the first 1,000 samples by 489.

[Nixon-hk]

This does look like a disk space issue. Note that disk requirements are different than RAM requirements. Imputing a half million samples will result in a very large output file. The exact size depends on the reference panel you are using. I would subset the samples instead of the region. I would create a VCF including only the first 1,000 samples and impute that. You can then extrapolate the total disk space needed by multiplying the output file size for the first 1,000 samples by 489.

Dear Jonathan,

Thank you so much for your insight and suggestions! I would look into it and try if that works.

Thanks again. Have a nice day.

Yours sincerely,

Nixon