Improved classification time with KMC

[dkoslicki]

When running StreamingQueryDNADatabase.py, in reality, we need only the K-mers in the sample that exist in the training database sketches. As such, it's possible to:

1. Dump all the training database sketch k-mers using KMC (like it's done here
2. Use KMC to count the k-mers in the sample
3. Intersect these with the k-mers in the training database sketches and dump these to a file
4. Reformat these dumped k-mers into a FASTA-looking file
5. Feed that into StreamingQueryDNADatabase.py

Steps 2-5 is basically what's done here as I noted this approach to Nathan LaPierre, but never got around to implementing it in CMash yet.

[dkoslicki]

Could be nice to write this as an optional script that will be run before the StreamingQueryDNADatabase.py script and then compare runtime performance with/without this pre-processing (as you’ll also not need the Bloom filter pre-filter). You’ll lose the ability to keep track of k-mer counts (which is the focus of #21), but it would be for a different use case (eg. Metalign). @IsaacT1123 This might be a good first thing to cut your teeth on and could be a component of the eventual CMash publication mentioned in #20 .

[IsaacT1123]

I’d be happy to work on this enhancement after getting CMash installed and creating a retraining script.

[dkoslicki]

Excellent! Go ahead and make a new branch (named something like KMC or Issue15 or the like), and I'll assign this issue to you

[dkoslicki]

@IsaacT1123 a friendly reminder to reference this issue in your commits! It's a great way to keep track of changes being made, and is also a visible history of your contributions to FOSS projects!

[IsaacT1123]

I’ll be sure to do so from now on- thank you for reminding me