Consolidate sparsity checks in CLI and CXG conversion steps.

[Bento007]

Motivation

We are running sparsity check in two place within our ingestion pipeline. Once during validation and again during cxg conversion. These are two separate algorithm doing the same thing. For efficiency they should be consolidated.

Definition of Done

• Consolidate the sparsity checks across the ingestion pipeline. Ideally it should only be calculated once and the result shared.

Tasks

• I recommend running the sparsity check in the CLI and passing the value to the CXG conversion step.

[nayib-jose-gloria]

Estimate (incl. testing): 2-3 days

@Bento007 for comment if you agree

[Bento007]

2-3 days if my suggested implementation is used.

[nayib-jose-gloria]

deferring this to DP maintenance work since it is not root of memory issues, as determined during investigation of #7310