Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parsing CWL v1.0 & v1.1 #103

Open
svonworl opened this issue Nov 8, 2022 · 8 comments
Open

parsing CWL v1.0 & v1.1 #103

svonworl opened this issue Nov 8, 2022 · 8 comments

Comments

@svonworl
Copy link

svonworl commented Nov 8, 2022

Recently, we integrated cwljava v1.0 into Dockstore, and during testing, we found some workflows that cause the parser to throw. In our webservice, we use our own preprocessor to combine the various component CWL files into one big CWL represented by Maps/Lists, and parse it with cwljava here:
https://github.com/dockstore/dockstore/blob/67f4547e771864cafacdc1c92fa7bd47261e32da/dockstore-webservice/src/main/java/io/dockstore/webservice/languages/CWLHandler.java#L364

The following workflows cause loadDocument to throw a ValidationException:

https://github.com/ICGC-TCGA-PanCancer/OxoG-Dockstore-Tools/tree/master
primary descriptor: /oxog_varbam_annotate_wf.cwl

https://github.com/h3abionet/h3agatk/tree/1.0.1
primary descriptor: /workflows/GATK/GATK-complete-WES-Workflow-h3abionet.cwl

The first workflow contains a SchemaDefRequirement and the parser appears to have trouble parsing the type references (TumourType.yaml#TumourType etc). When I change the type references to int, the workflow successfully parses.

Judging from exception message, the second workflow seems to be failing for a different reason, but I haven't pinpointed what, exactly. It is possible that it's not valid, but a cursory inspection didn't turn up any problems.

The exception messages are pretty big, so I put them and some stack trace info in the comments.

Please let me know if you need any more info. Thanks!

@svonworl
Copy link
Author

svonworl commented Nov 9, 2022

The first workflow exception message and stack trace is >600 lines long, below is an abbreviated version:

! org.w3id.cwl.cwl1_2.utils.ValidationException: Failed to match union type
!   Trying 'RecordField'
!     the `class` field is not valid because:
!       Expected one of [Ljava.lang.String;@270f831a
!     the `inputs` field is not valid because:
!       Failed to match union type
!         Expected object with Java type of java.util.List but got java.util.HashMap
!         Trying 'RecordField'
!           the `type` field is not valid because:
!             Failed to match union type
!               Expected object with Java type of java.lang.String but got java.util.LinkedHashMap
!               Expected object with Java type of java.lang.String but got java.util.LinkedHashMap
!               Trying 'RecordField'
!                 the `type` field is not valid because:
!                   Expected one of [Ljava.lang.String;@65553746
!               Trying 'RecordField'
!                 the `symbols` field is not valid because:
!                   Expected object with Java type of java.util.List but got null
!                 the `type` field is not valid because:
!                   Expected one of [Ljava.lang.String;@61bee517
!               Trying 'RecordField'

[  many lines deleted ]

!               Trying 'RecordField'
!                 the `symbols` field is not valid because:
!                   Expected object with Java type of java.util.List but got null
!                 the `type` field is not valid because:
!                   Expected one of [Ljava.lang.String;@61bee517
!               Trying 'RecordField'
!                 the `items` field is not valid because:
!                   TumourType.yaml#TumourType
!               Expected object with Java type of java.lang.String but got java.util.LinkedHashMap
!               Expected object with Java type of java.util.List but got java.util.LinkedHashMap
!   Expected object with Java type of java.util.List but got java.util.LinkedHashMap
! at org.w3id.cwl.cwl1_2.utils.UnionLoader.load(UnionLoader.java:31)
! at org.w3id.cwl.cwl1_2.utils.Loader.documentLoad(Loader.java:41)
! at org.w3id.cwl.cwl1_2.utils.RootLoader.loadDocument(RootLoader.java:18)

@svonworl
Copy link
Author

svonworl commented Nov 9, 2022

The second workflow exception message and stack trace:

! org.w3id.cwl.cwl1_2.utils.ValidationException: Failed to match union type
!   Trying 'RecordField'
!     the `class` field is not valid because:
!       Expected one of [Ljava.lang.String;@176252b7
!   Trying 'RecordField'
!     the `class` field is not valid because:
!       Expected one of [Ljava.lang.String;@6668dafa
!     the `expression` field is not valid because:
!       Expected a string.
!   Trying 'RecordField'
!     the `steps` field is not valid because:
!       Failed to match union type
!         Expected object with Java type of java.util.List but got java.util.HashMap
!         Trying 'RecordField'
!           the `run` field is not valid because:
!             Failed to match union type
!               Expected object with Java type of java.lang.String but got java.util.LinkedHashMap
!               Trying 'RecordField'
!                 the `class` field is not valid because:
!                   Expected one of [Ljava.lang.String;@176252b7
!               Trying 'RecordField'
!                 the `class` field is not valid because:
!                   Expected one of [Ljava.lang.String;@6668dafa
!                 the `expression` field is not valid because:
!                   Expected a string.
!               Trying 'RecordField'
!                 the `steps` field is not valid because:
!                   Failed to match union type
!                     Expected object with Java type of java.util.List but got java.util.HashMap
!                     Trying 'RecordField'
!                       the `run` field is not valid because:
!                         Failed to match union type
!                           Expected object with Java type of java.lang.String but got java.util.LinkedHashMap
!                           Trying 'RecordField'
!                             the `cwlVersion` field is not valid because:
!                               Expected one of [Ljava.lang.String;@5871ac54
!                           Trying 'RecordField'
!                             the `class` field is not valid because:
!                               Expected one of [Ljava.lang.String;@6668dafa
!                             the `cwlVersion` field is not valid because:
!                               Expected one of [Ljava.lang.String;@5871ac54
!                             the `expression` field is not valid because:
!                               Expected a string.
!                           Trying 'RecordField'
!                             the `class` field is not valid because:
!                               Expected one of [Ljava.lang.String;@13467c17
!                             the `cwlVersion` field is not valid because:
!                               Expected one of [Ljava.lang.String;@5871ac54
!                             the `steps` field is not valid because:
!                               Expected object with Java type of java.util.List but got null
!                           Trying 'RecordField'
!                             the `class` field is not valid because:
!                               Expected one of [Ljava.lang.String;@1f20d341
!                             the `cwlVersion` field is not valid because:
!                               Expected one of [Ljava.lang.String;@5871ac54
!               Trying 'RecordField'
!                 the `class` field is not valid because:
!                   Expected one of [Ljava.lang.String;@1f20d341
!   Trying 'RecordField'
!     the `class` field is not valid because:
!       Expected one of [Ljava.lang.String;@1f20d341
!   Expected object with Java type of java.util.List but got java.util.LinkedHashMap
! at org.w3id.cwl.cwl1_2.utils.UnionLoader.load(UnionLoader.java:31)
! at org.w3id.cwl.cwl1_2.utils.Loader.documentLoad(Loader.java:41)
! at org.w3id.cwl.cwl1_2.utils.RootLoader.loadDocument(RootLoader.java:18)

@mr-c
Copy link
Member

mr-c commented Nov 9, 2022

Hello @svonworl ; cwljava is currently only for CWL v1.2; if you'd like to parse CWL v1.1 and CWL v1.0 documents, that will have to be added as separate packages; you'll likely want code to dispatch to the correct loader based upon the cwlVersion

https://github.com/ICGC-TCGA-PanCancer/OxoG-Dockstore-Tools/blob/b38a8a4785746b8267913ea5389e21ae6dc921a3/oxog_varbam_annotate_wf.cwl#L3

@mr-c mr-c changed the title Some CWL workflows cause the parser to throw parsing CWL v1.0 & v1.1 Nov 9, 2022
@coverbeck
Copy link

@mr-c, are you planning to add support for 1.1 and 1.0? How much work would it be if we wanted to contribute that support ourselves (I haven't really looked at the codebase)?

@mr-c mr-c mentioned this issue Nov 10, 2022
3 tasks
@mr-c
Copy link
Member

mr-c commented Nov 10, 2022

@coverbeck I did a quick stab at CWL v1.1 in #105 (including a bit of documentation)

Would you like to take over the branch and continue with adding CWL v1.0 as well?

@coverbeck
Copy link

@mr-c , at this point we're still trying to figure out how/if we want to integrate cwljava with Dockstore. We couldn't commit to any work, at least not yet.

414 files in the PR! :) Presumably most of those are generated? Otherwise it seems like a massive undertaking. :)

@mr-c
Copy link
Member

mr-c commented Nov 10, 2022

@mr-c , at this point we're still trying to figure out how/if we want to integrate cwljava with Dockstore. We couldn't commit to any work, at least not yet.

414 files in the PR! :) Presumably most of those are generated? Otherwise it seems like a massive undertaking. :)

It is almost entirely automated, yes 😅

See https://github.com/common-workflow-language/cwljava/pull/105/files

@mr-c
Copy link
Member

mr-c commented Feb 16, 2023

Support for CWL v1.1 was added in #105 ; I can do the same for CWL v1.0, if someone requests it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants