crash: "The number of rows in 'eta' must be equal to the number of rows in 'counts'." when specifying a background argument

[plijnzaad]

Dear all,

if I run decontx (version 1.10.0) as

sce <- decontX(sce, batch=batch, z=labels, background=background)

I get:

Wed Mar  9 13:35:46 2022 ..  14  cells in the background matrix were removed as they were found in  the filtered matrix.
--------------------------------------------------
Starting DecontX
--------------------------------------------------
Wed Mar  9 13:35:47 2022 .. Analyzing cells in batch 'LX058'
Wed Mar  9 13:36:24 2022 .... Generating UMAP
Found more than one class "dist" in cache; using the first, from namespace 'BiocGenerics'
Also defined by 'spam'
Found more than one class "dist" in cache; using the first, from namespace 'BiocGenerics'
Also defined by 'spam'
Found more than one class "dist" in cache; using the first, from namespace 'BiocGenerics'
Also defined by 'spam'
Wed Mar  9 13:37:52 2022 .... Estimating contamination
Error in decontXLogLik(counts = counts, z = z, phi = phi, eta = eta, theta = theta,  : 
  The number of rows in 'eta' must be equal to the number of rows in 'counts'.

I think I specified everything in the right format:

> dim(background)
[1] 18554  7373
> dim(srat)
[1] 18554  7373
> identical(rownames(srat), rownames(background))
[1] TRUE
> is(background)
[1] "SingleCellExperiment"       "RangedSummarizedExperiment" "SummarizedExperiment"       "RectangularData"            "Vector"                     "Annotated"                  "vector_OR_Vector"          

PS:
I'm confused, when running decontx with a z-argument as per the above, I would have thought that no clustering is performed (since I supply that myself), so UMAP would not be needed, rigth? Yet I do see umap being run. Is that deliberate?

[plijnzaad]

For good measure, below is the code I used to create my background object, might shed light or be useful:

  libs <- c('LX058', 'LX060')
  for(lib in libs )  { 
    dir <- sprintf("/some/where/%s/raw_feature_bc_matrix", lib)
    tod  <- Seurat::Read10X(dir)
    colnames(tod) <- paste0(lib, '_', gsub("-1$", "", colnames(tod)))
    cols <- ncol(tod)
    cols <- setdiff(cols, colnames(srat)) #skip any non-empty droplets; srat is the Seurat object
    set.seed(cols)
    tod = tod[, sample(cols, size=sum(srat@meta.data$orig.ident==lib)) ]
    empties[[lib]] <- tod
  }
  tod <- do.call( cbind, empties)
  missing <- setdiff(rownames(srat), rownames(tod))
  m <- Matrix(data = 0, nrow = length(missing), ncol = ncol(tod), sparse = TRUE)
  rownames(m) <- missing
  colnames(m) <- colnames(tod)
  tod <- (rbind(tod, m))[ rownames(srat), ]
  background <- SingleCellExperiment(assays = list(counts = tod))
 

[joshua-d-campbell]

Hi @plijnzaad, thanks for using our tool! Note that the decontX function will remove any overlapping cell/barcode ids between the background/raw matrix and the cell/filtered matrix for you. So you don't need to do this yourself. I didn't go through your code in detail, but the number of columns in the background matrix is the same as the filtered matrix (i.e. 7373). Generally, the background matrix will have many more cell barcodes than the filtered matrix. If you want another quick way to import the raw and filtered data into an SCE object, you can use the importCellRanger from the singleCellTK package:

library(singleCellTK)
sce <- importCellRanger(sampleDirs = c("path/to/sample1/", "path/to/sample2/"))
sce.raw <- importCellRanger(sampleDirs = c("path/to/sample1/", "path/to/sample2/"), dataType = "raw")

Then you can run decontX directly from these:

sce <- decontX(sce,  z=labels, background=sce.raw)

I don't think the decontX function in v1.10.0 worked with batch and background parameters. So you may have to run it for each sample separately if using this version. We have updated this in the devel branch though and it will be released in the next Bioconductor release. You can install it with the following command:

library(devtools)
install_github("campbio/celda@devel")

Lastly, when supplying the z parameter, we still generate the UMAP for display in the downstream plots, but the cell clustering is skipped.

Hope that helps!

[plijnzaad]

Hi Joshua,

my question is about the crash, my code to specify the background is just to demonstrate (hopefully) that I set it up in the right 'shape'. Which it is , or not? So, how can I avoid this crash, or is it a bug?

Regarding the batch and background arguments, they are certainly there in v1.10.1, the helpfile says:

   batch: Numeric or character vector. Batch labels for cells. If batch
          labels are supplied, DecontX is run on cells from each batch
          separately. Cells run in different channels or assays should
          be considered different batches. Default NULL.

background: A numeric matrix of counts or a SingleCellExperiment with
          the matrix located in the assay slot under 'assayName'. It
          should have the same structure as 'x' except it contains the
          matrix of empty droplets instead of cells. When supplied,
          empirical distribution of transcripts from these empty
          droplets will be used as the contamination distribution.
          Default NULL.

I'll give the devel version a shot later. Cheers,
Philip

ps: my limiting the number of background droplets is to save memory, 3000 instead of 100,000 should still give a very good estimate

[joshua-d-campbell]

Yes, both parameters are present, but they don't work in conjunction with each other in v1.10.0 (i.e. the background matrix isn't properly slit up by sample/batch) and that may be what is contributing the error. So you can try running it separately for each sample or try the devel version in which batch/background will work with each other. Additionally, we really designed it to work with the whole background matrix so I don't think you should worry about subsampling.

[plijnzaad]

OK thanks!

[shabs24]

Hi,

Thanks for the tool. I am trying to run DecontX on one sample at a time and I am having the same error as above.
Here is the error

Error in decontXLogLik(counts = counts, z = z, phi = phi, eta = eta, theta = theta, :
The number of rows in 'eta' must be equal to the number of rows in 'counts’.

I am using version 1.14.2

Any suggestions?

[joshua-d-campbell]

Hi @shabs24, can you show the code you are using and give a sense of the size of the dataset? Also, can you show us the output of sessionInfo() so we can see what versions of dependencies you are using?

[shabs24]

Hi @joshua-d-campbell

Thanks for you quick response.

Here is the code and sessionInfo

Code

library(celda)
library(Seurat)

seuratObj=readRDS(filename)
sce=as.SingleCellExperiment(seuratObj)

counts.raw <- Read10X(rawfile)
sce.raw <- SingleCellExperiment(list(counts = counts.raw))

sce <- decontX(sce, background = sce.raw)

Error

Error in decontXLogLik(counts = counts, z = z, phi = phi, eta = eta, theta = theta, :
The number of rows in 'eta' must be equal to the number of rows in 'counts’.

SessionInfo

R version 4.2.1 (2022-06-23)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Monterey 12.6.2

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] Seurat_5.0.1                SeuratObject_5.0.1          sp_2.1-2                    celda_1.14.2               
 [5] Matrix_1.6-3                SingleCellExperiment_1.20.1 SummarizedExperiment_1.28.0 Biobase_2.58.0             
 [9] GenomicRanges_1.50.2        GenomeInfoDb_1.37.2         IRanges_2.32.0              S4Vectors_0.36.2           
[13] BiocGenerics_0.44.0         MatrixGenerics_1.10.0       matrixStats_1.1.0          

loaded via a namespace (and not attached):
  [1] spam_2.10-0               RcppEigen_0.3.3.9.4       plyr_1.8.9                igraph_1.5.1              assertive.files_0.0-2    
  [6] lazyeval_0.2.2            enrichR_3.2               splines_4.2.1             multipanelfigure_2.1.5    RcppHNSW_0.5.0           
 [11] BiocParallel_1.32.6       listenv_0.9.0             scattermore_1.2           scater_1.26.1             ggplot2_3.4.4            
 [16] digest_0.6.33             foreach_1.5.2             htmltools_0.5.7           viridis_0.6.4             magick_2.8.0             
 [21] fansi_1.0.5               magrittr_2.0.3            ScaledMatrix_1.6.0        assertive.numbers_0.0-2   tensor_1.5               
 [26] cluster_2.1.3             doParallel_1.0.17         ROCR_1.0-11               globals_0.16.2            R.utils_2.12.0           
 [31] spatstat.sparse_3.0-3     colorspace_2.1-0          ggrepel_0.9.4             WriteXLS_6.4.0            dplyr_1.1.4              
 [36] RCurl_1.98-1.13           jsonlite_1.8.7            progressr_0.14.0          spatstat.data_3.0-3       survival_3.3-1           
 [41] zoo_1.8-12                iterators_1.0.14          glue_1.6.2                polyclip_1.10-6           gtable_0.3.4             
 [46] zlibbioc_1.44.0           XVector_0.38.0            leiden_0.4.3.1            DelayedArray_0.24.0       BiocSingular_1.14.0      
 [51] future.apply_1.11.0       abind_1.4-5               scales_1.3.0              spatstat.random_3.2-1     miniUI_0.1.1.1           
 [56] Rcpp_1.0.11               viridisLite_0.4.2         xtable_1.8-4              gridGraphics_0.5-1        reticulate_1.34.0        
 [61] rsvd_1.0.5                dotCall64_1.1-1           htmlwidgets_1.6.3         httr_1.4.7                RColorBrewer_1.1-3       
 [66] ellipsis_0.3.2            ica_1.0-3                 scuttle_1.8.4             R.methodsS3_1.8.2         pkgconfig_2.0.3          
 [71] uwot_0.1.16               deldir_2.0-2              utf8_1.2.4                tidyselect_1.2.0          rlang_1.1.2              
 [76] reshape2_1.4.4            later_1.3.1               munsell_0.5.0             tools_4.2.1               cli_3.6.1                
 [81] dbscan_1.1-12             generics_0.1.3            ggridges_0.5.4            stringr_1.5.1             fastmap_1.1.1            
 [86] goftest_1.2-3             fitdistrplus_1.1-11       purrr_1.0.2               RANN_2.6.1                sparseMatrixStats_1.10.0 
 [91] nlme_3.1-158              pbapply_1.7-2             future_1.33.0             mime_0.12                 R.oo_1.25.0              
 [96] compiler_4.2.1            rstudioapi_0.13           beeswarm_0.4.0            plotly_4.10.3             curl_5.1.0               
[101] png_0.1-8                 spatstat.utils_3.0-4      tibble_3.2.1              stringi_1.8.2             RSpectra_0.16-1          
[106] lattice_0.20-45           assertive.base_0.0-9      vctrs_0.6.4               pillar_1.9.0              lifecycle_1.0.4          
[111] spatstat.geom_3.2-7       combinat_0.0-8            lmtest_0.9-40             BiocNeighbors_1.16.0      RcppAnnoy_0.0.21         
[116] data.table_1.14.8         cowplot_1.1.1             bitops_1.0-7              irlba_2.3.5.1             httpuv_1.6.12            
[121] patchwork_1.1.3           R6_2.5.1                  promises_1.2.1            KernSmooth_2.23-20        gridExtra_2.3            
[126] vipor_0.4.5               parallelly_1.36.0         codetools_0.2-18          MCMCprecision_0.4.0       fastDummies_1.7.3        
[131] MASS_7.3-58               rjson_0.2.21              withr_2.5.2               sctransform_0.4.1         GenomeInfoDbData_1.2.9   
[136] parallel_4.2.1            beachmat_2.14.2           grid_4.2.1                tidyr_1.3.0               DelayedMatrixStats_1.20.0
[141] Rtsne_0.16                spatstat.explore_3.2-5    shiny_1.8.0               ggbeeswarm_0.7.2         

[joshua-d-campbell]

Hi @shabs24, sorry for the delay. Can you try upgrading celda to the latest version? This can be done from Github directly using the command:

library(devtools)
install_github("campbio/celda")

Also, I'm not sure how well the as.SingleCellExperiment function works with Seurat v5 yet. Can you try creating a SingleCellExperiment object from the original count matrix. Hopefully the count matrix is stored as a "dgCMatrix". This can be checked by running class(counts(sce)).

Lastly, can you look at the dimensions of each matrix to make sure the number of rows agree:

dim(counts(sce))
dim(counts(sce.raw))

[batalha23]

Hello!
Thanks for the tool, and for all the suggestions already given to this issue. I have however also run into this same error and the previous solutions don't apply to my case (I believe), so I'd greatly appreciate any hint as to how to get past it :)

I want to run decontX on a dataset that is already stored in a Seurat object, so my steps were:

1. Load Seurat object
2. Convert Assay5 to Assay
3. Convert to SCE object
4. Import raw feature barcode matrix directly into an SCE object
5. Run decontX

I checked the dimensions of my main and reference (background) datasets and, as expected, they do have different numbers of genes and cells, which I believe is handled by decontX by removing overlapping barcode ids between the background/raw matrix and the cell/filtered matrix:

dim(DB006_01_SCE)
[1] 19114 13026
dim(DB006_01_ref)
[1] 33538 1673479

I am using decontX v1.0.0 due to Bioconductor compatibility with the rest of my packages for this project (R4.3.2, Bioconductor 3.18).

Below my code and session info. Thank you so much!

Code

#1. Load datasets
DB006_01_seu <- readRDS(file = 'path_to_file/DB006_01.rds')
DB006_01_seu[["RNA"]] <- as(object = DB006_01_seu[["RNA"]], Class = "Assay")

DB006_01_counts <- GetAssayData(object = DB006_01_seu, slot = "counts")
DB006_01_SCE <- SingleCellExperiment(list(counts = DB006_01_counts))

DB006_01_raw <- Read10X('data/raw/raw_feature_bc_matrix/DB006_01/')
DB006_01_ref <- SingleCellExperiment(list(counts = DB006_01_raw))

#2. Running decontX
DB006_01_SCE <- decontX(DB006_01_SCE, background = DB006_01_ref)

Error

Wed Oct 16 14:59:41 2024 ..  13026  cells in the background matrix were removed as they were found in  the filtered matrix.
--------------------------------------------------
Starting DecontX
--------------------------------------------------
Wed Oct 16 14:59:43 2024 .. Analyzing all cells
Wed Oct 16 14:59:43 2024 .... Generating UMAP and estimating cell types
Found more than one class "dist" in cache; using the first, from namespace 'spam'
Also defined by ‘BiocGenerics’
Found more than one class "dist" in cache; using the first, from namespace 'spam'
Also defined by ‘BiocGenerics’
Found more than one class "dist" in cache; using the first, from namespace 'spam'
Also defined by ‘BiocGenerics’
Wed Oct 16 15:00:45 2024 .... Estimating contamination
Error: The number of rows in 'eta' must be equal to the number of rows in 'counts'.

Session Info

R version 4.3.2 (2023-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 11 x64 (build 22631)

Matrix products: default


locale:
[1] LC_COLLATE=English_United Kingdom.utf8  LC_CTYPE=English_United Kingdom.utf8    LC_MONETARY=English_United Kingdom.utf8 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.utf8    

time zone: Europe/Lisbon
tzcode source: internal

attached base packages:
[1] stats4    stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] SingleCellExperiment_1.24.0 SummarizedExperiment_1.32.0 Biobase_2.62.0              GenomicRanges_1.54.1        GenomeInfoDb_1.38.8        
 [6] IRanges_2.36.0              S4Vectors_0.40.2            BiocGenerics_0.48.1         MatrixGenerics_1.14.0       matrixStats_1.2.0          
[11] decontX_1.0.0               BiocManager_1.30.22         viridis_0.6.5               viridisLite_0.4.2           RColorBrewer_1.1-3         
[16] xlsx_0.6.5                  cowplot_1.1.3               lubridate_1.9.3             forcats_1.0.0               stringr_1.5.1              
[21] purrr_1.0.2                 readr_2.1.5                 tidyr_1.3.1                 tibble_3.2.1                ggplot2_3.5.0              
[26] tidyverse_2.0.0             patchwork_1.2.0             Seurat_5.0.3                SeuratObject_5.0.1          sp_2.1-3                   
[31] dplyr_1.1.4                

loaded via a namespace (and not attached):
  [1] RcppAnnoy_0.0.22          splines_4.3.2             later_1.3.2               bitops_1.0-7              R.oo_1.26.0               polyclip_1.10-6          
  [7] fastDummies_1.7.3         lifecycle_1.0.4           StanHeaders_2.32.10       globals_0.16.3            lattice_0.21-9            MASS_7.3-60              
 [13] magrittr_2.0.3            plotly_4.10.4             yaml_2.3.8                httpuv_1.6.15             sctransform_0.4.1         spam_2.10-0              
 [19] pkgbuild_1.4.4            spatstat.sparse_3.0-3     reticulate_1.35.0         pbapply_1.7-2             abind_1.4-5               zlibbioc_1.48.2          
 [25] Rtsne_0.17                R.utils_2.12.3            RCurl_1.98-1.14           xlsxjars_0.6.1            GenomeInfoDbData_1.2.11   ggrepel_0.9.5            
 [31] inline_0.3.19             irlba_2.3.5.1             listenv_0.9.1             spatstat.utils_3.0-4      goftest_1.2-3             RSpectra_0.16-1          
 [37] spatstat.random_3.2-3     fitdistrplus_1.1-11       parallelly_1.37.1         DelayedMatrixStats_1.24.0 leiden_0.4.3.1            codetools_0.2-19         
 [43] DelayedArray_0.28.0       scuttle_1.12.0            tidyselect_1.2.1          farver_2.1.1              ScaledMatrix_1.10.0       spatstat.explore_3.2-7   
 [49] jsonlite_1.8.8            BiocNeighbors_1.20.2      progressr_0.14.0          scater_1.30.1             ggridges_0.5.6            survival_3.5-7           
 [55] dbscan_1.2-0              tools_4.3.2               ica_1.0-3                 Rcpp_1.0.12               glue_1.7.0                gridExtra_2.3            
 [61] SparseArray_1.2.4         loo_2.8.0                 withr_3.0.0               combinat_0.0-8            fastmap_1.1.1             MCMCprecision_0.4.0      
 [67] fansi_1.0.6               rsvd_1.0.5                digest_0.6.35             timechange_0.3.0          R6_2.5.1                  mime_0.12                
 [73] colorspace_2.1-0          scattermore_1.2           tensor_1.5                spatstat.data_3.0-4       R.methodsS3_1.8.2         utf8_1.2.4               
 [79] generics_0.1.3            renv_1.0.3                data.table_1.15.4         httr_1.4.7                htmlwidgets_1.6.4         S4Arrays_1.2.1           
 [85] uwot_0.1.16               pkgconfig_2.0.3           rJava_1.0-11              gtable_0.3.4              lmtest_0.9-40             XVector_0.42.0           
 [91] htmltools_0.5.8           dotCall64_1.1-1           scales_1.3.0              png_0.1-8                 rstudioapi_0.16.0         tzdb_0.4.0               
 [97] reshape2_1.4.4            nlme_3.1-163              zoo_1.8-12                KernSmooth_2.23-22        vipor_0.4.7               parallel_4.3.2           
[103] miniUI_0.1.1.1            pillar_1.9.0              grid_4.3.2                vctrs_0.6.5               RANN_2.6.1                promises_1.2.1           
[109] BiocSingular_1.18.0       beachmat_2.18.1           xtable_1.8-4              cluster_2.1.4             beeswarm_0.4.0            cli_3.6.2                
[115] compiler_4.3.2            rlang_1.1.3               crayon_1.5.2              rstantools_2.4.0          future.apply_1.11.2       labeling_0.4.3           
[121] ggbeeswarm_0.7.2          plyr_1.8.9                stringi_1.8.3             rstan_2.32.6              BiocParallel_1.36.0       deldir_2.0-4             
[127] QuickJSR_1.4.0            munsell_0.5.0             lazyeval_0.2.2            spatstat.geom_3.2-9       Matrix_1.6-5              RcppHNSW_0.6.0           
[133] hms_1.1.3                 sparseMatrixStats_1.14.0  future_1.33.2             shiny_1.8.1               ROCR_1.0-11               igraph_2.0.3             
[139] RcppParallel_5.1.9

[joshua-d-campbell]

Hi @batalha23, thanks for posting this issue and sending the matrix dimensions. It looks like you have different numbers of genes (i.e. rows) between your filtered expression matrix and your raw background matrix. This may happen as a result of getting the filtered expression matrix from the Seurat object where some genes have been filtered out. DecontX requires the same number of genes in each matrix. We will try to make this error more explicit in the future.

[CampingOnSeabed]

Hi @batalha23, thanks for posting this issue and sending the matrix dimensions. It looks like you have different numbers of genes (i.e. rows) between your filtered expression matrix and your raw background matrix. This may happen as a result of getting the filtered expression matrix from the Seurat object where some genes have been filtered out. DecontX requires the same number of genes in each matrix. We will try to make this error more explicit in the future.

Hello!
Thank you for developing this tool! I met a similar problem saying that "The number of rows in 'eta' must be equal to the number of rows in 'counts'", and I realized that this is due to the different gene number bewteen my sce_count and background_raw_count. The sce_count was from the filtered_feature_bc_matrix of the output of cellranger, and background_raw_count was from the raw_feature_bc_matrix of the output of cellranger. I didn't do any change to the gene number. So how can I fix the error? can I exclude the genes in background_raw_count which are not in the gene matrix of my seurat data to get the same gene number?
Best wish