From d4d18717c41e2677c2e7214b8d4dc0e0fb0b4e5b Mon Sep 17 00:00:00 2001 From: Fienne Date: Thu, 29 Dec 2022 21:09:15 +0100 Subject: [PATCH 01/12] add info on how outputBinding and glob works --- src/faq.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/faq.md b/src/faq.md index 50843d6a..ece0e101 100644 --- a/src/faq.md +++ b/src/faq.md @@ -447,6 +447,9 @@ The reference runner and several other CWL implementations support running those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. +## How outputBinding.glob works +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory ` Wildcard characters can be used in the filenames when searching for files. + ## Debug JavaScript Expressions You can use the --js-console option of cwltool, or you can try From 890274329dd7ebfb5a9d25d59bc50ddc43ed45c4 Mon Sep 17 00:00:00 2001 From: Fienne Date: Thu, 29 Dec 2022 21:16:56 +0100 Subject: [PATCH 02/12] add wildcard characters --- src/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/faq.md b/src/faq.md index ece0e101..6fd2f608 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,7 +448,7 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory ` Wildcard characters can be used in the filenames when searching for files. +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*` or a question mark `?`. ## Debug JavaScript Expressions From 70b85c07ff258adea22e9e8f8508d9e9f99edeaf Mon Sep 17 00:00:00 2001 From: Fienne Date: Thu, 29 Dec 2022 21:21:35 +0100 Subject: [PATCH 03/12] add wildcard example --- src/faq.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/faq.md b/src/faq.md index 6fd2f608..23ca1446 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,7 +448,25 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*` or a question mark `?`. +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*` or a question mark `?`. + +```cwl +outputs: + example_out: + type: File + outputBinding: + glob: hello.txt +``` + +The glob field in the above output section can also be written with the `*` wildcard character as + +```cwl +outputs: + example_out: + type: File + outputBinding: + glob: '*.txt' +``` ## Debug JavaScript Expressions From 4f4c763f64e2dddd2f0c3af199a55390cd818b87 Mon Sep 17 00:00:00 2001 From: Fienne Date: Fri, 6 Jan 2023 14:47:26 +0100 Subject: [PATCH 04/12] more info on glob --- src/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/faq.md b/src/faq.md index 23ca1446..5fb729bc 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,7 +448,7 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all file paths that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*` or a question mark `?`. +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*`, a question mark `?` or a range `[]` ```cwl outputs: From d6e514209cb260465ba4b87662dc6a5dd0bd14b6 Mon Sep 17 00:00:00 2001 From: Fienne Date: Mon, 9 Jan 2023 13:38:31 +0100 Subject: [PATCH 05/12] more info on glob --- src/faq.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/faq.md b/src/faq.md index 5fb729bc..f6d7caee 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,7 +448,9 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The value of the glob field is the `output directory` Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*`, a question mark `?` or a range `[]` +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The value of the glob field is the `output directory`. +Glob's pattern rules abide to standard Unix path expansion rules. According to benchmarks, it is also expected to be faster than other methods for matching pathnames in directories. +Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*`, a question mark matching pathnames in directories. `?` or a range `[]` ```cwl outputs: @@ -456,7 +458,7 @@ outputs: type: File outputBinding: glob: hello.txt -``` +``` The glob field in the above output section can also be written with the `*` wildcard character as From 6ecaecd9c1c0785918749c3e1e64f4acb5b287b8 Mon Sep 17 00:00:00 2001 From: Fienne Date: Wed, 11 Jan 2023 07:14:46 +0100 Subject: [PATCH 06/12] more info on glob and examples --- src/faq.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/src/faq.md b/src/faq.md index f6d7caee..489719a7 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,10 +448,13 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The value of the glob field is the `output directory`. +The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The `glob` field specifies the file and the output directory in which the file should be stored. The value of the glob field is the `output directory`. +If the value of glob is a relative path pattern (one that does not start with a slash '/'), it is resolved relative to the output directory. If the glob value is an absolute path pattern (it starts with a slash '/'), it must refer to a path within the output directory. Glob's pattern rules abide to standard Unix path expansion rules. According to benchmarks, it is also expected to be faster than other methods for matching pathnames in directories. Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*`, a question mark matching pathnames in directories. `?` or a range `[]` +In the example below, the glob field is used to specify the file(`hello.txt`) in which the output should be stored. + ```cwl outputs: example_out: @@ -470,6 +473,24 @@ outputs: glob: '*.txt' ``` +Glob can also be used to return all outputs from a CommandLine Tool. For example + +``` + outputs: + type: Directory + outputBinding: { glob: . } +``` + +OR + +``` + outputs: + type: + type: array + items: [Directory, File] + outputBinding: {glob: "*"} +``` + ## Debug JavaScript Expressions You can use the --js-console option of cwltool, or you can try From 9c9123d8b8a9c98a64a3c93363a732815b4ed467 Mon Sep 17 00:00:00 2001 From: Fienne Date: Mon, 23 Jan 2023 12:35:29 +0100 Subject: [PATCH 07/12] edit information on glob --- src/faq.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/faq.md b/src/faq.md index 489719a7..b9832acc 100644 --- a/src/faq.md +++ b/src/faq.md @@ -447,11 +447,12 @@ The reference runner and several other CWL implementations support running those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. -## How outputBinding.glob works -The `outputBinding` section describes how to set the value of each output parameter. The `glob` field is used to return all files/pathnames that match a specific pattern. The `glob` field specifies the file and the output directory in which the file should be stored. The value of the glob field is the `output directory`. -If the value of glob is a relative path pattern (one that does not start with a slash '/'), it is resolved relative to the output directory. If the glob value is an absolute path pattern (it starts with a slash '/'), it must refer to a path within the output directory. -Glob's pattern rules abide to standard Unix path expansion rules. According to benchmarks, it is also expected to be faster than other methods for matching pathnames in directories. -Wildcard characters can be used in the filenames when searching for files. The wildcard characters can either be an asterisk `*`, a question mark matching pathnames in directories. `?` or a range `[]` +## How does glob work when describing output values? +The field outputBinding describes how to to set the value of each output parameter. The 'glob' field specifies a pattern to find files/directories relative to the output directory. The pattern used for glob' must be either relative to the output directory, an absolute path to the output directory, or an absolute path to an input file. + +CWL uses the POSIX glob(3) pathname matching. Wildcards are allowed in the glob field and are useful when If you don’t know the exact name of a file or directory in advance. The wildcard characters can either be an asterisk *, a question mark matching pathnames in directories. `?` or a range `[]`. + +If an array used in the glob field, any files that match any pattern in the array are returned. In the example below, the glob field is used to specify the file(`hello.txt`) in which the output should be stored. @@ -473,7 +474,7 @@ outputs: glob: '*.txt' ``` -Glob can also be used to return all outputs from a CommandLine Tool. For example +Glob can also be used to return all files in the output directory. For example ``` outputs: From 003bb64729e9253e15615b595c77053e4d344e32 Mon Sep 17 00:00:00 2001 From: Fienne Date: Mon, 23 Jan 2023 12:49:11 +0100 Subject: [PATCH 08/12] edit glob example --- src/faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/faq.md b/src/faq.md index b9832acc..495c9011 100644 --- a/src/faq.md +++ b/src/faq.md @@ -448,7 +448,7 @@ those Docker format containers using the Singularity engine. Directly specifying a Singularity format container is not part of the CWL standards. ## How does glob work when describing output values? -The field outputBinding describes how to to set the value of each output parameter. The 'glob' field specifies a pattern to find files/directories relative to the output directory. The pattern used for glob' must be either relative to the output directory, an absolute path to the output directory, or an absolute path to an input file. +The field outputBinding describes how to set the value of each output parameter. The 'glob' field specifies a pattern to find files/directories relative to the output directory. The pattern used for glob' must be either relative to the output directory, an absolute path to the output directory, or an absolute path to an input file. CWL uses the POSIX glob(3) pathname matching. Wildcards are allowed in the glob field and are useful when If you don’t know the exact name of a file or directory in advance. The wildcard characters can either be an asterisk *, a question mark matching pathnames in directories. `?` or a range `[]`. @@ -461,7 +461,7 @@ outputs: example_out: type: File outputBinding: - glob: hello.txt + glob: "hello.txt" ``` The glob field in the above output section can also be written with the `*` wildcard character as From 47ec637c4e513a2845e282021d0671f5e07ef9c8 Mon Sep 17 00:00:00 2001 From: Fienne Date: Mon, 6 Feb 2023 10:35:29 -0800 Subject: [PATCH 09/12] modify info on glob and examples --- src/faq.md | 47 +++++++++++++++-------------------------------- 1 file changed, 15 insertions(+), 32 deletions(-) diff --git a/src/faq.md b/src/faq.md index 495c9011..37cf38b3 100644 --- a/src/faq.md +++ b/src/faq.md @@ -454,43 +454,26 @@ CWL uses the POSIX glob(3) pathname matching. Wildcards are allowed in the glob If an array used in the glob field, any files that match any pattern in the array are returned. -In the example below, the glob field is used to specify the file(`hello.txt`) in which the output should be stored. +In the example below, the glob field is used to return all outputs from the tool. -```cwl -outputs: - example_out: - type: File - outputBinding: - glob: "hello.txt" ``` - -The glob field in the above output section can also be written with the `*` wildcard character as - -```cwl -outputs: - example_out: +cwlVersion: v1.0 +class: CommandLineTool +baseCommand: [tar, --extract] +inputs: + tarfile: type: File - outputBinding: - glob: '*.txt' -``` - -Glob can also be used to return all files in the output directory. For example - -``` - outputs: - type: Directory - outputBinding: { glob: . } + inputBinding: + prefix: --file +outputs: + my_output: + type: + type: array + items: [Directory, File] + outputBinding: + glob: "*" ``` -OR - -``` - outputs: - type: - type: array - items: [Directory, File] - outputBinding: {glob: "*"} -``` ## Debug JavaScript Expressions From cc2792bdcfda4a97ca47c594f233f1482375b616 Mon Sep 17 00:00:00 2001 From: Fienne Date: Mon, 6 Feb 2023 10:44:34 -0800 Subject: [PATCH 10/12] modify info on glob and examples --- src/faq.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/faq.md b/src/faq.md index 37cf38b3..82b0ff38 100644 --- a/src/faq.md +++ b/src/faq.md @@ -457,14 +457,19 @@ If an array used in the glob field, any files that match any pattern in the arra In the example below, the glob field is used to return all outputs from the tool. ``` -cwlVersion: v1.0 class: CommandLineTool -baseCommand: [tar, --extract] +cwlVersion: v1.0 inputs: - tarfile: + in1: type: File + default: + class: File + path: /path/to/my/file inputBinding: - prefix: --file + position: 1 + +baseCommand: cat + outputs: my_output: type: From b420ea5dba7e6cf53b09b5cd4eab18075529c590 Mon Sep 17 00:00:00 2001 From: Ngumih Fien Date: Wed, 22 Feb 2023 12:40:42 -0800 Subject: [PATCH 11/12] add glob example --- src/faq.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/src/faq.md b/src/faq.md index 82b0ff38..a997096a 100644 --- a/src/faq.md +++ b/src/faq.md @@ -450,15 +450,15 @@ specifying a Singularity format container is not part of the CWL standards. ## How does glob work when describing output values? The field outputBinding describes how to set the value of each output parameter. The 'glob' field specifies a pattern to find files/directories relative to the output directory. The pattern used for glob' must be either relative to the output directory, an absolute path to the output directory, or an absolute path to an input file. -CWL uses the POSIX glob(3) pathname matching. Wildcards are allowed in the glob field and are useful when If you don’t know the exact name of a file or directory in advance. The wildcard characters can either be an asterisk *, a question mark matching pathnames in directories. `?` or a range `[]`. +CWL uses the POSIX glob(3) pathname matching. Wildcards are allowed in the glob field and are useful when If you don’t know the exact name of a file or directory in advance. The wildcard characters can either be an asterisk `*`, a question mark matching pathnames in directories. `?` or a range `[]`. If an array used in the glob field, any files that match any pattern in the array are returned. -In the example below, the glob field is used to return all outputs from the tool. +In the example below, the glob field using the `*` wildcard is used to return all outputs from the tool. ``` -class: CommandLineTool cwlVersion: v1.0 +class: CommandLineTool inputs: in1: type: File @@ -479,6 +479,24 @@ outputs: glob: "*" ``` +In the following example, the wildcard `?` is used in the glob field to return all tools with the letter `a` in their pathname. + +``` +cwlVersion: v1.0 +class: CommandLineTool +baseCommand: [tar, --extract] +inputs: + tarfile: + type: File + inputBinding: + prefix: --file +outputs: + untarfile: + type: File[] + outputBinding: + glob: "?a" + +``` ## Debug JavaScript Expressions From 2e8d3db2b751cc421c4ff7f076f2d90b80a5ccb6 Mon Sep 17 00:00:00 2001 From: Ngumih Fien Date: Tue, 7 Mar 2023 08:14:22 -0800 Subject: [PATCH 12/12] add glob example --- src/faq.md | 40 ++++++++++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/src/faq.md b/src/faq.md index a997096a..6aa65b54 100644 --- a/src/faq.md +++ b/src/faq.md @@ -479,23 +479,43 @@ outputs: glob: "*" ``` -In the following example, the wildcard `?` is used in the glob field to return all tools with the letter `a` in their pathname. - +Below is an example where the input file is used as the output file using `glob` ``` +#!/usr/bin/env cwl-runner cwlVersion: v1.0 class: CommandLineTool -baseCommand: [tar, --extract] + +doc: | + Merge multiple BAM files. + +hints: + ResourceRequirement: + coresMin: 1 + ramMin: 20000 + DockerRequirement: + dockerPull: quay.io/biocontainers/samtools:1.14--hb421002_0 + +baseCommand: ["samtools", "merge"] + inputs: - tarfile: - type: File + - id: output_name + doc: name of merged bam file + type: string + inputBinding: + position: 1 + - id: bams + doc: bam files to be merged + type: + type: array + items: File inputBinding: - prefix: --file + position: 2 + outputs: - untarfile: - type: File[] + - id: bam_merged + type: File outputBinding: - glob: "?a" - + glob: $(inputs.output_name) ``` ## Debug JavaScript Expressions