Define Galaxy API for sending tracks when launching Galaxy

[MillenniumFalconMechanic]

Need

We have the API to list the tracks for an assembly but assuming the user selected multiple tracks, how do we send the track URLs when launching Galaxy?

[mvdbeek]

Same as we do now,

brc-analytics/app/utils/galaxy-api.ts

Line 82 in 6bff3c8

"Annotation GTF": { ext: "gtf.gz", src: "url", url: geneModelUrl },

. I'll make it so that this can be set up in workflows.yml (xref #342).

There is however a missing piece in that the workflows can't do anything with track data, we need one of the common genomic formats, such as gtf, gff, bed, bigwig etc.

[mvdbeek]

ping @maximilianh @NullModel, is there a way we can get those common genomic formats on the fly ? Can we know what representation and encoding makes sense for the data ? If we can do the conversion, will we get the data back out lossless and are the attributes and relationships preserved in a way that e.g rnaseq analysis will work properly, i.e gene ids, children, transcript ids, promoter exon etc etc

[maximilianh]

what do you mean with "lossless" exactly? We cannot use GFF as-is, as it's too slow, so we convert that to an internal format and given how verbose and vague the GFF spec is, it's not possible to re-create the original GFF file entirely, with all the dozens of strange fields that no one uses (e.g. the CIGAR strings, or the IDs used for linking things together, which are only used for linking). We use bigGenePred internally (which is just a bigBed with a few more fields) which should convert just fine back to GFF. Most RNA-seq workflows that I know only need to have the exons and the transcript ID. Do you have specific requirements? But we also keep the original GFF from what I know, would you prefer that? Everything else is either a bigBed or a bigWig so that should work but you won't be using these anyways, right? So this ticket can probably be more concretely phrased as "do the GFFs produced by our converter work in Galaxy suggested's RNA-seq pipeline" ? I hope it does. If it doesn't, then we will fix that.

…

On Tue, Mar 4, 2025 at 3:58 AM Marius van den Beek ***@***.***> wrote: ping @maximilianh <https://github.com/maximilianh> @NullModel <https://github.com/NullModel>, is there a way we can get those common genomic formats on the fly ? Can we know what representation and encoding makes sense for the data ? If we can do the conversion, will we get the data back out lossless and are the attributes and relationships preserved in a way that e.g rnaseq analysis will work properly, i.e gene ids, children, transcript ids, promoter exon etc etc — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TIWP4WETRSJAQLQSFD2SVTLTAVCNFSM6AAAAABYH7YH7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWGY4TQNJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: mvdbeek]*mvdbeek* left a comment (galaxyproject/brc-analytics#348) <#348 (comment)> ping @maximilianh <https://github.com/maximilianh> @NullModel <https://github.com/NullModel>, is there a way we can get those common genomic formats on the fly ? Can we know what representation and encoding makes sense for the data ? If we can do the conversion, will we get the data back out lossless and are the attributes and relationships preserved in a way that e.g rnaseq analysis will work properly, i.e gene ids, children, transcript ids, promoter exon etc etc — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACL4TIWP4WETRSJAQLQSFD2SVTLTAVCNFSM6AAAAABYH7YH7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWGY4TQNJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[maximilianh]

Would it be helpful if we send you the command to convert our format back to GFF? It's something like bigGenePredToGenePred and then genePredToGtf, but we can test this. Is there a way for us to check if the produced GFF fulfills your requirements? On Tue, Mar 4, 2025 at 6:54 AM Maximilian Haeussler ***@***.***> wrote:

…

what do you mean with "lossless" exactly? We cannot use GFF as-is, as it's too slow, so we convert that to an internal format and given how verbose and vague the GFF spec is, it's not possible to re-create the original GFF file entirely, with all the dozens of strange fields that no one uses (e.g. the CIGAR strings, or the IDs used for linking things together, which are only used for linking). We use bigGenePred internally (which is just a bigBed with a few more fields) which should convert just fine back to GFF. Most RNA-seq workflows that I know only need to have the exons and the transcript ID. Do you have specific requirements? But we also keep the original GFF from what I know, would you prefer that? Everything else is either a bigBed or a bigWig so that should work but you won't be using these anyways, right? So this ticket can probably be more concretely phrased as "do the GFFs produced by our converter work in Galaxy suggested's RNA-seq pipeline" ? I hope it does. If it doesn't, then we will fix that. On Tue, Mar 4, 2025 at 3:58 AM Marius van den Beek < ***@***.***> wrote: > ping @maximilianh <https://github.com/maximilianh> @NullModel > <https://github.com/NullModel>, is there a way we can get those common > genomic formats on the fly ? Can we know what representation and encoding > makes sense for the data ? If we can do the conversion, will we get the > data back out lossless and are the attributes and relationships preserved > in a way that e.g rnaseq analysis will work properly, i.e gene ids, > children, transcript ids, promoter exon etc etc > > — > Reply to this email directly, view it on GitHub > <#348 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACL4TIWP4WETRSJAQLQSFD2SVTLTAVCNFSM6AAAAABYH7YH7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWGY4TQNJSGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > [image: mvdbeek]*mvdbeek* left a comment > (galaxyproject/brc-analytics#348) > <#348 (comment)> > > ping @maximilianh <https://github.com/maximilianh> @NullModel > <https://github.com/NullModel>, is there a way we can get those common > genomic formats on the fly ? Can we know what representation and encoding > makes sense for the data ? If we can do the conversion, will we get the > data back out lossless and are the attributes and relationships preserved > in a way that e.g rnaseq analysis will work properly, i.e gene ids, > children, transcript ids, promoter exon etc etc > > — > Reply to this email directly, view it on GitHub > <#348 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACL4TIWP4WETRSJAQLQSFD2SVTLTAVCNFSM6AAAAABYH7YH7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWGY4TQNJSGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[mvdbeek]

what do you mean with "lossless" exactly?

Ideally we'd get back out whatever was submitted, that way there's less room for bugs to creep in. If we can say "report it at the source" should there be faulty annotations our lives would certainly be easier.

But we also keep the original GFF from what I
know, would you prefer that?

💯 absolutely, but I suppose that's not what this issue is about ? In fact, @MillenniumFalconMechanic @nekrut, what's the user going to do with the track data ? Is that the usecase where the user just selects data and then launches things manually in Galaxy ?

Do you have specific requirements?

No, other than that the RNAseq and variant calling workflows consume GTF files. I suppose a side-by-side comparison with iGenomes (i'm under the impression that serves as the gold standard?) for one or two of the host organisms that are represented in BRC and iGenomes is probably a good idea to make sure we got all that we need.

Would it be helpful if we send you the command to convert our format back
to GFF? It's something like bigGenePredToGenePred and then genePredToGtf,

GTF is what we consume right now, it would definitely be helpful to have the commands. I'm hoping we won't have to prefix our workflows with conversion steps, that's why I was asking if you'd be open to streaming these out from your API ?

Is there a way for us to check if the produced GFF fulfills your requirements?

As you said, that's hard to know in a general way. I think we can only work usecase by usecase (i.e workflow by workflow).
I see mockups elsewhere in this repo that include repeat tracks and so on, and that has me wondering what exactly the plan is for these "tracks".

[maximilianh]

Ideally we'd get back out whatever was submitted, that way there's less room for bugs to creep in. If we can say "report it at the source" should there be faulty annotations our lives would certainly be easier.

GFF is a very special format, one were every conversion will produce a different file (as you probably know).

Do you have specific requirements? No, other than that the RNAseq and variant calling workflows consume GTF files.

Our GTF files are a lot cleaner than the NCBI ones.

I suppose a side-by-side comparison with iGenomes (i'm under the impression that servers as the gold standard?) for one or two of the host organisms that are represented in BRC and iGenomes is probably a good idea to make sure we got all that we need.

iGenomes uses GTF. Not the huge NCBI GFFs. At least the tarball I just checked.

Would it be helpful if we send you the command to convert our format back to GFF? It's something like bigGenePredToGenePred and then genePredToGtf, GTF is what we consume right now, it would definitely be helpful to have the commands. I'm hoping we won't have to prefix our workflows with conversion steps, that's why I was asking if you'd be open to streaming these out from your API ?

I'd rather not do on-the-fly conversions. What we can do is to prepare GTF files for you. Actually, that's what we have already, here, and I think Hiram pointed this out, e.g. here https://hgdownload.soe.ucsc.edu/hubs/GCF/000/298/275/GCF_000298275.1/genes/ The GTF is minimal and has all the required fields, transcriptId, gene symbol and exon information, it is a GTF and that may be better than the overly verbose NCBI ones: NC_002078.1 ncbiRefSeq.2022-12-19 transcript 15386 15452 . - . gene_id "rna-NC_002078.1:15386..15452"; transcript_id "rna-NC_002078.1:15386..15452"; gene_name "rna-NC_002078.1:15386..15452"; NC_002078.1 ncbiRefSeq.2022-12-19 exon 15386 15452 . - . gene_id "rna-NC_002078.1:15386..15452"; transcript_id "rna-NC_002078.1:15386..15452"; exon_number "1"; exon_id "rna-NC_002078.1:15386..15452.1"; gene_name "rna-NC_002078.1:15386..15452"; NC_002078.1 ncbiRefSeq.2022-12-19 transcript 15311 15377 . + . gene_id "rna-NC_002078.1:15311..15377"; transcript_id "rna-NC_002078.1:15311..15377"; gene_name "rna-NC_002078.1:15311..15377"; NC_002078.1 ncbiRefSeq.2022-12-19 exon 15311 15377 . + . gene_id "rna-NC_002078.1:15311..15377"; transcript_id "rna-NC_002078.1:15311..15377"; exon_number "1"; exon_id "rna-NC_002078.1:15311..15377.1"; gene_name "rna-NC_002078.1:15311..15377"; NC_002078.1 ncbiRefSeq.2022-12-19 transcript 14171 15310 . + . gene_id "CYTB"; transcript_id "NP_008587.1"; gene_name "CYTB";

Is there a way for us to check if the produced GFF fulfills your requirements? As you said, that's hard to know in a general way. I think we can only work usecase by usecase (i.e workflow by workflow). I see mockups elsewhere in this repo that include repeat tracks and so on, and that has me wondering what exactly the plan is for these "tracks".

Sorry I don't understand this sentence. What do you mean with "plan" and why is "tracks" in quotes - are repeats not annotation tracks? best Max

…

Message ID: ***@***.***>

[mvdbeek]

Hiram pointed this out, e.g. here

He was also saying that we shouldn't use them ... I might not have gotten the finer points there. I agree that we can use them, that's what we currently do.

Sorry I don't understand this sentence. What do you mean with "plan" and
why is "tracks" in quotes - are repeats not annotation tracks?

What is the plan for the user to do with the track data they select ("tracks" because that's in the issue title), which determines in what format we need the data. repeats are tracks, but what is a user going to do with them ?

[maximilianh]

He was also saying that we shouldn't use them ... I might not have gotten the finer points there. I agree that we can use them, that's what we currently do.

@NullModel why should they not use the files that you made?

What is the plan for the user to do with the track data they select ("tracks" because that's in the issue title), which determines in what format we need the data. repeats are tracks, but what is a user going to do with them ?

I don't know. What is the use case / the requirements here? I don't know what users could do with repeatmasker tracks.

…

Message ID: ***@***.***>

[maximilianh]

Users send the repeatmasker track content to Galaxy sometimes to do intersections. E.g. "which genes have repeats in their introns" or "which promoters contain repeat X". These types of questions. But I guess that's not part of the BRC workflows. I will talk to Anton tonight about what the context here is for BRC.

…

On Tue, Mar 4, 2025 at 11:18 AM Maximilian Haeussler ***@***.***> wrote: He was also saying that we shouldn't use them ... I might not have gotten > the finer points there. I agree that we can use them, that's what we > currently do. > @NullModel why should they not use the files that you made? > What is the plan for the user to do with the track data they select > ("tracks" because that's in the issue title), which determines in what > format we need the data. repeats are tracks, but what is a user going to do > with them ? > I don't know. What is the use case / the requirements here? I don't know what users could do with repeatmasker tracks. > Message ID: ***@***.*** > com> >