diff --git a/docs/process-typed.md b/docs/process-typed.md index e26fadfd0a..55c2778a69 100644 --- a/docs/process-typed.md +++ b/docs/process-typed.md @@ -65,6 +65,8 @@ All {ref}`standard types ` except for the dataflow types (`Channel Nextflow automatically stages `Path` inputs and `Path` collections (such as `Set`) into the task directory. +### Nullable inputs + By default, tasks fail if any input receives a `null` value. To allow `null` values, add `?` to the type annotation: ```nextflow @@ -85,10 +87,58 @@ process cat_opt { } ``` -### Stage directives +### Record inputs + +Inputs with type `Record` can declare the name and type of each record field: + +```nextflow +process fastqc { + input: + (id: String, fastq: Path): Record + + script: + """ + echo 'id: ${id}` + echo 'fastq: ${fastq}' + """ +} +``` + +This pattern is called *record destructuring*. Each record field is staged into the task the same way as an individual input. + +When the process is invoked, the incoming record should contain the specified fields, or else the run will fail. If the record has additional fields not declared by the process input, they are ignored. + +:::{tip} +Record inputs are a useful way to select a subset of fields from a larger record. This way, the process only stages what it needs, allowing you to keep related data together in your workflow logic. +::: + +### Tuple inputs + +Inputs with type `Tuple` can declare the name of each tuple component: + +```nextflow +process fastqc { + input: + (id, fastq): Tuple + + script: + """ + echo 'id: ${id}` + echo 'fastq: ${fastq}' + """ +} +``` + +This pattern is called *tuple destructuring*. Each tuple component is staged into the task the same way as an individual input. + +The generic types inside the `Tuple<...>` annotation specify the type of each tuple compomnent and should match the component names. In the above example, `id` has type `String` and `fastq` has type `Path`. + +## Stage directives The `stage:` section defines custom staging behavior using *stage directives*. It should be specified after the `input:` section. These directives serve the same purpose as input qualifiers such as `env` and `stdin` in the legacy syntax. +### Environment variables + The `env` directive declares an environment variable in terms of task inputs: ```nextflow @@ -106,6 +156,8 @@ process echo_env { } ``` +### Standard input (stdin) + The `stdin` directive defines the standard input of the task script: ```nextflow @@ -123,6 +175,8 @@ process cat { } ``` +### Custom file staging + The `stageAs` directive stages an input file (or files) under a custom file pattern: ```nextflow @@ -222,6 +276,40 @@ process foo { } ``` +### Structured outputs + +Whereas legacy process outputs could only be structured using specific qualifiers like `val` and `tuple`, typed process outputs are regular values. + +The `record()` standard library function can be used to create a record: + +```nextflow +process fastqc { + input: + (id: String, fastq: Path): Record + + output: + record(id: id, fastqc: file('fastqc_logs')) + + script: + // ... +} +``` + +The `tuple()` standard library function can be used to create a tuple: + +```nextflow +process fastqc { + input: + (id, fastq): Tuple + + output: + tuple(id, file('fastqc_logs')) + + script: + // ... +} +``` + ## Topics The `topic:` section emits values to {ref}`topic channels `. A topic emission consists of an output value and a topic name: diff --git a/docs/reference/stdlib-namespaces.md b/docs/reference/stdlib-namespaces.md index 6d422eee23..972d7a8bb1 100644 --- a/docs/reference/stdlib-namespaces.md +++ b/docs/reference/stdlib-namespaces.md @@ -108,8 +108,11 @@ The global namespace contains globally available constants and functions. `sleep( milliseconds: long )` : Sleep for the given number of milliseconds. +`record( [options] ) -> Record` +: Create a record from the given named arguments. + `tuple( args... ) -> Tuple` -: Create a tuple object from the given arguments. +: Create a tuple from the given arguments. (stdlib-namespaces-channel)= diff --git a/docs/reference/stdlib-types.md b/docs/reference/stdlib-types.md index 5d46dca09f..36aa8475a6 100644 --- a/docs/reference/stdlib-types.md +++ b/docs/reference/stdlib-types.md @@ -767,6 +767,38 @@ The following methods are available for splitting and counting the records in fi `splitText() -> List` : Splits a text file into a list of lines. See the {ref}`operator-splittext` operator for available options. +(stdlib-types-record)= + +## Record + +A record is an immutable map of fields to values (i.e., `Map`). Each value can have its own type. + +A record can be created using the `record` function: + +```nextflow +sample = record(id: '1', fastq_1: file('1_1.fastq'), fastq_2: file('1_2.fastq')) +``` + +Record fields can be accessed as properties: + +```nextflow +sample.id +// -> '1' +``` + +The following operations are supported for records: + +`+ : (Record, Record) -> Record` +: Given two records, returns a new record containing the fields and values of both records. When a field is present in both records, the value of the right-hand record takes precedence. + +`- : (Record, Iterable) -> Record` +: Given a record and a collection of strings, returns a copy of the record with the given fields removed. + +The following methods are available for a record: + +`subMap( keys: Iterable ) -> Record` +: Returns a new record containing only the given fields. + (stdlib-types-set)= ## Set\ diff --git a/docs/reference/syntax.md b/docs/reference/syntax.md index 8d14cc22ec..0c63ac6e6a 100644 --- a/docs/reference/syntax.md +++ b/docs/reference/syntax.md @@ -34,6 +34,7 @@ A Nextflow script may contain the following top-level declarations: - Process definitions - Function definitions - Enum types +- Record types - Output block Script declarations are in turn composed of statements and expressions. @@ -360,9 +361,17 @@ enum Day { Enum values in the above example can be accessed as `Day.MONDAY`, `Day.TUESDAY`, and so on. -:::{note} -Enum types cannot be included across modules at this time. -::: +### Record type + +A record type declaration consists of a name and a body. The body consists of one or more fields, where each field has a name and a type: + +```nextflow +record FastqPair { + id: String + fastq_1: Path + fastq_2: Path +} +``` ### Output block diff --git a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy index 96283d5352..bdd76db910 100644 --- a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy @@ -37,6 +37,7 @@ import nextflow.splitter.FastaSplitter import nextflow.splitter.FastqSplitter import nextflow.util.ArrayTuple import nextflow.util.CacheHelper +import nextflow.util.RecordMap import org.slf4j.Logger import org.slf4j.LoggerFactory /** @@ -150,6 +151,14 @@ class Nextflow { return result instanceof Collection ? result : [result] } + /** + * Creates a {@link RecordMap} from the given named arguments. + * + * @param props + */ + static RecordMap record(Map props) { + return new RecordMap(props) + } /** * Creates a {@link ArrayTuple} object with the given open array items diff --git a/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy b/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy index 4ebfa77807..47568b7f47 100644 --- a/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy @@ -1769,7 +1769,7 @@ class TaskProcessor { if( value == null && !param.optional ) throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${i} cannot be null -- append `?` to the type annotation to mark it as nullable") if( param instanceof ProcessTupleInput ) - assignTaskTupleInput(task, param, value, i) + assignTaskStructuredInput(task, param, value, i) else assignTaskInput(task, param, value, i) } @@ -1806,17 +1806,26 @@ class TaskProcessor { } @CompileStatic - private void assignTaskTupleInput(TaskRun task, ProcessTupleInput param, Object value, int index) { - if( value !instanceof List ) { - throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple but received: ${value} [${value.class.simpleName}]") + private void assignTaskStructuredInput(TaskRun task, ProcessTupleInput param, Object value, int index) { + if( value instanceof List ) { + final tupleParams = param.getComponents() + final tupleValues = value as List + if( tupleParams.size() != tupleValues.size() ) { + throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple with ${tupleParams.size()} elements but received ${tupleValues.size()} -- offending value: $tupleValues") + } + for( int i = 0; i < tupleParams.size(); i++ ) { + assignTaskInput(task, tupleParams[i], tupleValues[i], index) + } } - final tupleParams = param.getComponents() - final tupleValues = value as List - if( tupleParams.size() != tupleValues.size() ) { - throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a tuple with ${tupleParams.size()} elements but received ${tupleValues.size()} -- offending value: $tupleValues") + else if( value instanceof Map ) { + final record = value as Map + for( final fieldParam : param.getComponents() ) { + final fieldName = fieldParam.getName() + assignTaskInput(task, fieldParam, record[fieldName], index) + } } - for( int i = 0; i < tupleParams.size(); i++ ) { - assignTaskInput(task, tupleParams[i], tupleValues[i], index) + else { + throw new ProcessUnrecoverableException("[${safeTaskName(task)}] input at index ${index} expected a record or tuple but received: ${value} [${value.class.simpleName}]") } } diff --git a/modules/nf-commons/src/main/nextflow/util/HashBuilder.java b/modules/nf-commons/src/main/nextflow/util/HashBuilder.java index a753f368e4..6d6b73097b 100644 --- a/modules/nf-commons/src/main/nextflow/util/HashBuilder.java +++ b/modules/nf-commons/src/main/nextflow/util/HashBuilder.java @@ -27,6 +27,7 @@ import java.nio.file.attribute.BasicFileAttributes; import java.util.Collection; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.Set; import java.util.UUID; @@ -46,7 +47,6 @@ import nextflow.extension.Bolts; import nextflow.extension.FilesEx; import nextflow.io.SerializableMarker; -import nextflow.script.types.Bag; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import static nextflow.Const.DEFAULT_ROOT; @@ -141,14 +141,21 @@ else if( value instanceof CharSequence ) else if( value instanceof byte[] ) hasher.putBytes( (byte[])value ); - else if( value instanceof Object[]) + else if( value instanceof Object[]) { for( Object item : ((Object[])value) ) with(item); + } - // note: should map be order invariant as Set ? - else if( value instanceof Map ) - for( Object item : ((Map)value).values() ) + else if( value instanceof CacheFunnel ) + ((CacheFunnel)value).funnel(hasher, mode); + + else if( value instanceof List ) { + for( Object item : ((List)value) ) with(item); + } + + else if( value instanceof Map ) + hashUnorderedCollection(hasher, ((Map) value).entrySet(), mode); else if( value instanceof Map.Entry ) { Map.Entry entry = (Map.Entry)value; @@ -156,13 +163,9 @@ else if( value instanceof Map.Entry ) { with(entry.getValue()); } - else if( value instanceof Bag || value instanceof Set ) + else if( value instanceof Collection ) hashUnorderedCollection(hasher, (Collection) value, mode); - else if( value instanceof Collection) - for( Object item : ((Collection)value) ) - with(item); - else if( value instanceof Path ) hashFile(hasher, (Path)value, mode, basePath); @@ -180,9 +183,6 @@ else if( value instanceof VersionNumber ) else if( value instanceof SerializableMarker) hasher.putInt( value.hashCode() ); - else if( value instanceof CacheFunnel ) - ((CacheFunnel)value).funnel(hasher, mode); - else if( value instanceof Enum ) hasher.putUnencodedChars( value.getClass().getName() + "." + value ); diff --git a/modules/nf-commons/src/main/nextflow/util/RecordMap.java b/modules/nf-commons/src/main/nextflow/util/RecordMap.java new file mode 100644 index 0000000000..b8e238a6ba --- /dev/null +++ b/modules/nf-commons/src/main/nextflow/util/RecordMap.java @@ -0,0 +1,67 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.util; + +import java.util.Collection; +import java.util.HashMap; +import java.util.Map; + +import nextflow.script.types.Record; +import org.codehaus.groovy.runtime.DefaultGroovyMethods; + +/** + * Implements Record as an immutable map. + * + * @author Ben Sherman + */ +public class RecordMap extends HashMap implements Record { + + public RecordMap() {} + + public RecordMap(Map props) { + super(props); + } + + public RecordMap(int initialCapacity) { + super(initialCapacity); + } + + @Override + public void clear() { + throw new UnsupportedOperationException(); + } + + @Override + public Object put(String key, Object value) { + throw new UnsupportedOperationException(); + } + + // @Override + // public void putAll(Map m) { + // throw new UnsupportedOperationException(); + // } + + @Override + public Object remove(Object key) { + throw new UnsupportedOperationException(); + } + + public Record subMap(Collection keys) { + return new RecordMap(DefaultGroovyMethods.subMap(this, keys)); + } + +} diff --git a/modules/nf-lang/src/main/antlr/ScriptLexer.g4 b/modules/nf-lang/src/main/antlr/ScriptLexer.g4 index bb465b33ed..268c7c2beb 100644 --- a/modules/nf-lang/src/main/antlr/ScriptLexer.g4 +++ b/modules/nf-lang/src/main/antlr/ScriptLexer.g4 @@ -312,7 +312,7 @@ NEW : 'new'; // PRIVATE : 'private'; // PROTECTED : 'protected'; // PUBLIC : 'public'; -// RECORD : 'record'; +RECORD : 'record'; RETURN : 'return'; // SEALED : 'sealed'; diff --git a/modules/nf-lang/src/main/antlr/ScriptParser.g4 b/modules/nf-lang/src/main/antlr/ScriptParser.g4 index 96fd876c23..4cd33a4d67 100644 --- a/modules/nf-lang/src/main/antlr/ScriptParser.g4 +++ b/modules/nf-lang/src/main/antlr/ScriptParser.g4 @@ -112,6 +112,7 @@ scriptDeclaration | importDeclaration #importDeclAlt | paramsDef #paramsDefAlt | paramDeclarationV1 #paramDeclV1Alt + | recordDef #recordDefAlt | enumDef #enumDefAlt | processDef #processDefAlt | workflowDef #workflowDefAlt @@ -169,6 +170,17 @@ paramDeclarationV1 : PARAMS (DOT identifier)+ nls ASSIGN nls expression ; +// -- record definition +recordDef + : RECORD identifier nls LBRACE + nls recordBody? + nls RBRACE + ; + +recordBody + : nameTypePair (sep nameTypePair)* + ; + // -- enum definition enumDef : ENUM identifier nls LBRACE @@ -228,7 +240,7 @@ processInputs processInput : identifier (COLON type)? - | LPAREN identifier (COMMA identifier)+ rparen (COLON type)? + | LPAREN nameTypePair (COMMA nameTypePair)* rparen (COLON type)? | statement ; @@ -576,6 +588,7 @@ identifier | NEXTFLOW | PARAMS | FROM + | RECORD | PROCESS | EXEC | INPUT @@ -778,6 +791,7 @@ keywords | PARAMS | INCLUDE | FROM + | RECORD | PROCESS | EXEC | INPUT diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/IncludeEntryNode.java b/modules/nf-lang/src/main/java/nextflow/script/ast/IncludeEntryNode.java index 5e1b132053..92f9a7d6a6 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/IncludeEntryNode.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/IncludeEntryNode.java @@ -16,7 +16,7 @@ package nextflow.script.ast; import org.codehaus.groovy.ast.ASTNode; -import org.codehaus.groovy.ast.MethodNode; +import org.codehaus.groovy.ast.AnnotatedNode; /** * An included process, workflow, or function. @@ -40,13 +40,13 @@ public String getNameOrAlias() { return alias != null ? alias : name; } - private MethodNode target; + private AnnotatedNode target; - public void setTarget(MethodNode target) { + public void setTarget(AnnotatedNode target) { this.target = target; } - public MethodNode getTarget() { + public AnnotatedNode getTarget() { return target; } } diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/RecordNode.java b/modules/nf-lang/src/main/java/nextflow/script/ast/RecordNode.java new file mode 100644 index 0000000000..b4c1180934 --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/RecordNode.java @@ -0,0 +1,34 @@ +/* + * Copyright 2024-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.script.ast; + +import java.lang.reflect.Modifier; + +import nextflow.script.types.Record; +import org.codehaus.groovy.ast.ClassHelper; +import org.codehaus.groovy.ast.ClassNode; + +/** + * A record type definition. + * + * @author Ben Sherman + */ +public class RecordNode extends ClassNode { + public RecordNode(String name) { + super(name, Modifier.PUBLIC | Modifier.FINAL, ClassHelper.OBJECT_TYPE); + setInterfaces(new ClassNode[] { ClassHelper.makeCached(Record.class) }); + } +} diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptNode.java b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptNode.java index 230b51feef..980c58880e 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptNode.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptNode.java @@ -110,7 +110,7 @@ public List getFunctions() { public List getTypes() { return getClasses().stream() - .filter(cn -> cn.isEnum()) + .filter(cn -> cn instanceof RecordNode || cn.isEnum()) .toList(); } diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitor.java index 7650c25823..47cf14185e 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitor.java @@ -43,6 +43,8 @@ public interface ScriptVisitor extends GroovyCodeVisitor { void visitFunction(FunctionNode node); + void visitRecord(RecordNode node); + void visitEnum(ClassNode node); void visitOutputs(OutputBlockNode node); diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitorSupport.java b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitorSupport.java index c9b0608455..28afd9c13f 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitorSupport.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/ScriptVisitorSupport.java @@ -43,7 +43,9 @@ public void visit(ScriptNode script) { for( var functionNode : script.getFunctions() ) visitFunction(functionNode); for( var classNode : script.getClasses() ) { - if( classNode.isEnum() ) + if( classNode instanceof RecordNode rn ) + visitRecord(rn); + else if( classNode.isEnum() ) visitEnum(classNode); } if( script.getOutputs() != null ) @@ -117,6 +119,12 @@ public void visitFunction(FunctionNode node) { visit(node.getCode()); } + @Override + public void visitRecord(RecordNode node) { + for( var fn : node.getFields() ) + visitField(fn); + } + @Override public void visitEnum(ClassNode node) { for( var fn : node.getFields() ) diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/TupleParameter.java b/modules/nf-lang/src/main/java/nextflow/script/ast/TupleParameter.java index 28e2c088a3..6df215c648 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/TupleParameter.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/TupleParameter.java @@ -21,8 +21,8 @@ import org.codehaus.groovy.ast.Parameter; /** - * A parameter that destructures the components of a tuple - * by name. + * A parameter that destructures the components of a record + * or tuple by name. * * @author Ben Sherman */ @@ -33,4 +33,9 @@ public TupleParameter(ClassNode type, Parameter[] components) { super(type, ""); this.components = components; } + + public boolean isRecord() { + return "Record".equals(getType().getUnresolvedName()) + || "Record".equals(getType().getNameWithoutPackage()); + } } diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java index 31eb801bd6..f4f0da398a 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java @@ -27,6 +27,8 @@ import nextflow.script.ast.ScriptNode; import nextflow.script.ast.ScriptVisitorSupport; import org.codehaus.groovy.ast.ASTNode; +import org.codehaus.groovy.ast.AnnotatedNode; +import org.codehaus.groovy.ast.ClassNode; import org.codehaus.groovy.ast.MethodNode; import org.codehaus.groovy.control.SourceUnit; import org.codehaus.groovy.control.messages.SyntaxErrorMessage; @@ -102,7 +104,7 @@ public void visitInclude(IncludeNode node) { for( var entry : node.entries ) { var includedName = entry.name; var includedNode = definitions.stream() - .filter(defNode -> includedName.equals(defNode.getName())) + .filter(defNode -> includedName.equals(definitionName(defNode))) .findFirst(); if( !includedNode.isPresent() ) { addError("Included name '" + includedName + "' is not defined in module '" + includeUri.getPath() + "'", node); @@ -140,15 +142,23 @@ private boolean isIncludeStale(IncludeNode node, URI includeUri) { return false; } - private List getDefinitions(URI uri) { + private List getDefinitions(URI uri) { var scriptNode = (ScriptNode) compiler.getSource(uri).getAST(); - var result = new ArrayList(); + var result = new ArrayList(); result.addAll(scriptNode.getWorkflows()); result.addAll(scriptNode.getProcesses()); result.addAll(scriptNode.getFunctions()); + result.addAll(scriptNode.getTypes()); return result; } + private static String definitionName(AnnotatedNode node) { + return + node instanceof ClassNode cn ? cn.getName() : + node instanceof MethodNode mn ? mn.getName() : + null; + } + @Override public void addError(String message, ASTNode node) { var cause = new ResolveIncludeError(message, node); diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveVisitor.java index 6fe6445df0..e8b5b9a6ba 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveVisitor.java @@ -69,6 +69,7 @@ public class ResolveVisitor extends ClassCodeExpressionTransformer { ClassHelper.LIST_TYPE, ClassHelper.MAP_TYPE, ClassHelper.makeCached(java.nio.file.Path.class), + ClassHelper.makeCached(nextflow.script.types.Record.class), ClassHelper.SET_TYPE, ClassHelper.STRING_TYPE, ClassHelper.makeCached(nextflow.script.types.Tuple.class) diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ScriptResolveVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ScriptResolveVisitor.java index 2d93018daa..8eb8f7424c 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ScriptResolveVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ScriptResolveVisitor.java @@ -24,10 +24,13 @@ import nextflow.script.ast.ParamNodeV1; import nextflow.script.ast.ProcessNodeV1; import nextflow.script.ast.ProcessNodeV2; +import nextflow.script.ast.RecordNode; import nextflow.script.ast.ScriptNode; import nextflow.script.ast.ScriptVisitorSupport; +import nextflow.script.ast.TupleParameter; import nextflow.script.ast.WorkflowNode; import org.codehaus.groovy.ast.ClassNode; +import org.codehaus.groovy.ast.FieldNode; import org.codehaus.groovy.ast.DynamicVariable; import org.codehaus.groovy.ast.Parameter; import org.codehaus.groovy.ast.expr.VariableExpression; @@ -67,6 +70,8 @@ public void visit() { var variableScopeVisitor = new VariableScopeVisitor(sourceUnit); variableScopeVisitor.declare(); variableScopeVisitor.visit(); + + // TOOD: enable resolve visitor to use included types // resolve type names if( sn.getParams() != null ) @@ -79,6 +84,12 @@ public void visit() { visitProcess(processNode); for( var functionNode : sn.getFunctions() ) visitFunction(functionNode); + for( var type : sn.getClasses() ) { + if( type instanceof RecordNode rn ) + visitRecord(rn); + else if( type.isEnum() ) + visitEnum(type); + } if( sn.getOutputs() != null ) visitOutputs(sn.getOutputs()); @@ -126,8 +137,13 @@ private void resolveTypedOutputs(Statement block) { @Override public void visitProcessV2(ProcessNodeV2 node) { - for( var input : node.inputs ) + for( var input : node.inputs ) { resolver.resolveOrFail(input.getType(), input); + if( input instanceof TupleParameter tp && tp.isRecord() ) { + for( var component : tp.components ) + resolver.resolveOrFail(component.getType(), component); + } + } resolver.visit(node.directives); resolver.visit(node.stagers); resolveTypedOutputs(node.outputs); @@ -158,6 +174,11 @@ public void visitFunction(FunctionNode node) { resolver.visit(node.getCode()); } + @Override + public void visitField(FieldNode node) { + resolver.resolveOrFail(node.getType(), node); + } + @Override public void visitOutput(OutputNode node) { resolver.resolveOrFail(node.getType(), node.getType()); diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java index 4d3fd88a70..01f7f9497b 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java @@ -33,7 +33,6 @@ import nextflow.script.ast.ProcessNodeV2; import nextflow.script.ast.ScriptNode; import nextflow.script.ast.ScriptVisitorSupport; -import nextflow.script.ast.TupleParameter; import nextflow.script.ast.WorkflowNode; import nextflow.script.dsl.Constant; import nextflow.script.dsl.EntryWorkflowDsl; @@ -114,18 +113,19 @@ public void declare() { declareMethod(processNode); for( var functionNode : sn.getFunctions() ) declareMethod(functionNode); + declareTypes(sn); } } private void declareInclude(IncludeNode node) { for( var entry : node.entries ) { - if( entry.getTarget() == null ) - continue; - var name = entry.getNameOrAlias(); - var otherInclude = vsc.getInclude(name); - if( otherInclude != null ) - vsc.addError("`" + name + "` is already included", node, "First included here", otherInclude); - vsc.include(name, entry.getTarget()); + if( entry.getTarget() instanceof MethodNode mn ) { + var name = entry.getNameOrAlias(); + var otherInclude = vsc.getInclude(name); + if( otherInclude != null ) + vsc.addError("`" + name + "` is already included", node, "First included here", otherInclude); + vsc.include(name, mn); + } } } @@ -170,6 +170,20 @@ private void declareMethod(MethodNode mn) { cn.addMethod(mn); } + private void declareTypes(ScriptNode sn) { + var types = sn.getTypes(); + for( int i = 0; i < types.size(); i++ ) { + // TODO: check included types + for( int j = 0; j < i; j++ ) { + var first = types.get(j); + var second = types.get(i); + if( !first.getName().equals(second.getName()) ) + continue; + vsc.addError("`" + second.getName() + "` is already declared", second, "First declared here", first); + } + } + } + public void visit() { var moduleNode = sourceUnit.getAST(); if( moduleNode instanceof ScriptNode sn ) { diff --git a/modules/nf-lang/src/main/java/nextflow/script/dsl/ScriptDsl.java b/modules/nf-lang/src/main/java/nextflow/script/dsl/ScriptDsl.java index 49db52f7ed..3a0bbaae53 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/dsl/ScriptDsl.java +++ b/modules/nf-lang/src/main/java/nextflow/script/dsl/ScriptDsl.java @@ -27,6 +27,7 @@ import nextflow.script.namespaces.LogNamespace; import nextflow.script.namespaces.NextflowNamespace; import nextflow.script.namespaces.WorkflowNamespace; +import nextflow.script.types.Record; import nextflow.script.types.Tuple; /** @@ -194,6 +195,11 @@ Collection files( """) void println(Object value); + @Description(""" + Create a record from the given named arguments. + """) + Record record(Map opts); + @Description(""" Send an email. """) diff --git a/modules/nf-lang/src/main/java/nextflow/script/formatter/ScriptFormattingVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/formatter/ScriptFormattingVisitor.java index ed45c8f896..1f56e9b291 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/formatter/ScriptFormattingVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/formatter/ScriptFormattingVisitor.java @@ -32,11 +32,14 @@ import nextflow.script.ast.ProcessNode; import nextflow.script.ast.ProcessNodeV1; import nextflow.script.ast.ProcessNodeV2; +import nextflow.script.ast.RecordNode; import nextflow.script.ast.ScriptNode; import nextflow.script.ast.ScriptVisitorSupport; import nextflow.script.ast.TupleParameter; import nextflow.script.ast.WorkflowNode; +import nextflow.script.types.Types; import org.codehaus.groovy.ast.ClassNode; +import org.codehaus.groovy.ast.FieldNode; import org.codehaus.groovy.ast.Parameter; import org.codehaus.groovy.ast.expr.EmptyExpression; import org.codehaus.groovy.ast.expr.Expression; @@ -131,6 +134,8 @@ else if( decl instanceof ParamNodeV1 pn ) visitParamV1(pn); else if( decl instanceof ProcessNode pn ) visitProcess(pn); + else if( decl instanceof RecordNode rn ) + visitRecord(rn); else if( decl instanceof WorkflowNode wn ) visitWorkflow(wn); } @@ -314,12 +319,13 @@ private void visitTypedInputs(Parameter[] inputs) { for( var input : inputs ) { fmt.appendIndent(); if( input instanceof TupleParameter tp ) { + var components = Arrays.stream(tp.components) + .map(p -> ( + tp.isRecord() ? p.getName() + ": " + Types.getName(p.getType()) : p.getName() + )) + .collect(Collectors.joining(", ")); fmt.append('('); - fmt.append( - Arrays.stream(tp.components) - .map(p -> p.getName()) - .collect(Collectors.joining(", ")) - ); + fmt.append(components); fmt.append(')'); } else { @@ -576,6 +582,45 @@ public void visitFunction(FunctionNode node) { fmt.append("}\n"); } + @Override + public void visitRecord(RecordNode node) { + fmt.appendLeadingComments(node); + fmt.append("record "); + fmt.append(node.getName()); + visitRecordBody(node); + } + + private void visitRecordBody(RecordNode node) { + var alignmentWidth = options.harshilAlignment() + ? maxFieldWidth(node.getFields()) + : 0; + + fmt.append(" {\n"); + fmt.incIndent(); + for( var fn : node.getFields() ) { + fmt.appendIndent(); + fmt.append(fn.getName()); + if( fmt.hasType(fn) ) { + if( alignmentWidth > 0 ) { + var padding = alignmentWidth - fn.getName().length() + 1; + fmt.append(" ".repeat(padding)); + } + fmt.append(": "); + fmt.visitTypeAnnotation(fn.getType()); + } + fmt.appendNewLine(); + } + fmt.decIndent(); + fmt.appendIndent(); + fmt.append("}\n"); + } + + private int maxFieldWidth(List fields) { + return fields.stream() + .map(fn -> fn.getName().length()) + .max(Integer::compare).orElse(0); + } + @Override public void visitEnum(ClassNode node) { fmt.appendLeadingComments(node); diff --git a/modules/nf-lang/src/main/java/nextflow/script/parser/ScriptAstBuilder.java b/modules/nf-lang/src/main/java/nextflow/script/parser/ScriptAstBuilder.java index 1f5c517b12..dd205d7a8a 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/parser/ScriptAstBuilder.java +++ b/modules/nf-lang/src/main/java/nextflow/script/parser/ScriptAstBuilder.java @@ -42,6 +42,7 @@ import nextflow.script.ast.ProcessNode; import nextflow.script.ast.ProcessNodeV1; import nextflow.script.ast.ProcessNodeV2; +import nextflow.script.ast.RecordNode; import nextflow.script.ast.ScriptNode; import nextflow.script.ast.TupleParameter; import nextflow.script.ast.WorkflowNode; @@ -66,6 +67,7 @@ import org.codehaus.groovy.ast.ASTNode; import org.codehaus.groovy.ast.ClassHelper; import org.codehaus.groovy.ast.ClassNode; +import org.codehaus.groovy.ast.FieldNode; import org.codehaus.groovy.ast.GenericsType; import org.codehaus.groovy.ast.ModuleNode; import org.codehaus.groovy.ast.NodeMetaDataHandler; @@ -317,6 +319,12 @@ else if( ctx instanceof ProcessDefAltContext pdac ) { moduleNode.addProcess(node); } + else if( ctx instanceof RecordDefAltContext rdac ) { + var node = recordDef(rdac.recordDef()); + saveLeadingComments(node, ctx); + moduleNode.addClass(node); + } + else if( ctx instanceof WorkflowDefAltContext wdac ) { var node = workflowDef(wdac.workflowDef()); saveLeadingComments(node, ctx); @@ -410,6 +418,33 @@ private IncludeNode includeDeclaration(IncludeDeclarationContext ctx) { return ast( new IncludeNode(source, entries), ctx ); } + private RecordNode recordDef(RecordDefContext ctx) { + var name = identifier(ctx.identifier()); + var result = ast( new RecordNode(name), ctx ); + if( ctx.recordBody() != null ) + recordBody(ctx.recordBody(), result); + else + collectSyntaxError(new SyntaxException("Missing record body", result)); + groovydocManager.handle(result, ctx); + return result; + } + + private void recordBody(RecordBodyContext ctx, RecordNode result) { + for( var el : ctx.nameTypePair() ) { + var param = nameTypePair(el); + if( el.type() == null ) + collectSyntaxError(new SyntaxException("Missing field type", param)); + var fn = ast( new FieldNode( + param.getText(), + Modifier.PUBLIC, + param.getType(), + result, + null), el ); + groovydocManager.handle(fn, el); + result.addField(fn); + } + } + private ClassNode enumDef(EnumDefContext ctx) { var name = identifier(ctx.identifier()); var result = ast( EnumHelper.makeEnumNode(name, Modifier.PUBLIC, null, null), ctx ); @@ -496,40 +531,76 @@ private Parameter processInput(ProcessInputContext ctx) { collectSyntaxError(new SyntaxException("Invalid input declaration in typed process", result)); return null; } + var type = type(ctx.type()); - var names = ctx.identifier().stream().map(this::identifier).toList(); - var result = names.size() == 1 - ? ast( param(type, names.get(0)), ctx ) - : processTupleInput(type, names, ctx); - for( var name : names ) + if( ctx.identifier() != null ) { + var name = identifier(ctx.identifier()); + var result = ast( param(type, name), ctx ); checkInvalidVarName(name, result); - if( names.size() == 1 && ctx.type() == null ) - collectWarning("Process input should have a type annotation", names.get(0), result); - saveTrailingComment(result, ctx); - return result; + if( ctx.type() == null ) + collectWarning("Process input should have a type annotation", name, result); + saveTrailingComment(result, ctx); + return result; + } + + if( "Record".equals(type.getUnresolvedName()) ) { + var result = processRecordInput(type, ctx); + saveTrailingComment(result, ctx); + return result; + } + + if( "Tuple".equals(type.getUnresolvedName()) ) { + var result = processTupleInput(type, ctx); + saveTrailingComment(result, ctx); + return result; + } + + var result = ast( new EmptyStatement(), ctx ); + collectSyntaxError(new SyntaxException("Destructured process input should have type Record or Tuple", result)); + return null; + } + + private TupleParameter processRecordInput(ClassNode type, ProcessInputContext ctx) { + var components = ctx.nameTypePair().stream() + .map((ntp) -> { + var name = identifier(ntp.identifier()); + var componentType = type(ntp.type()); + var component = ast( param(componentType, name), ntp ); + checkInvalidVarName(name, component); + if( ntp.type() == null ) + collectWarning("Record component should have a type annotation", component.getName(), component); + return component; + }) + .toArray(Parameter[]::new); + return ast( new TupleParameter(type, components), ctx ); } - private TupleParameter processTupleInput(ClassNode type, List names, ProcessInputContext ctx) { - var componentTypes = tupleComponentTypes(type, names.size()); - var components = new Parameter[names.size()]; - for( int i = 0; i < names.size(); i++ ) { + private TupleParameter processTupleInput(ClassNode type, ProcessInputContext ctx) { + var numComponents = ctx.nameTypePair().size(); + var componentTypes = tupleComponentTypes(type, numComponents); + var components = new Parameter[numComponents]; + for( int i = 0; i < numComponents; i++ ) { + var ntp = ctx.nameTypePair().get(i); + var name = identifier(ntp.identifier()); var componentType = componentTypes != null ? componentTypes.get(i) : ClassHelper.dynamicType(); - components[i] = ast( param(componentType, names.get(i)), ctx.identifier().get(i) ); + var component = ast( param(componentType, name), ntp.identifier() ); + checkInvalidVarName(component.getName(), component); + if( ntp.type() != null ) + collectWarning("Tuple component should not have a type annotation", component.getName(), component); + components[i] = component; } var result = ast( new TupleParameter(type, components), ctx ); - if( !"Tuple".equals(type.getUnresolvedName()) ) - collectSyntaxError(new SyntaxException("Process tuple input must have type `Tuple<...>`", result)); - if( !type.isUsingGenerics() || type.getGenericsTypes().length != names.size() ) - collectSyntaxError(new SyntaxException("Process tuple input type must have " + names.size() + " type arguments (one for each tuple component)", result)); + if( !type.isUsingGenerics() || type.getGenericsTypes().length != numComponents ) + collectSyntaxError(new SyntaxException("Process tuple input type must have " + numComponents + " type arguments (one for each tuple component)", result)); + if( components.length == 1 ) + collectSyntaxError(new SyntaxException("Process tuple input type more than one component", result)); return result; } private List tupleComponentTypes(ClassNode type, int n) { if( !"Tuple".equals(type.getUnresolvedName()) ) return null; - if( !type.isUsingGenerics() ) - return null; - if( type.getGenericsTypes().length != n ) + if( !type.isUsingGenerics() || type.getGenericsTypes().length != n ) return null; return Arrays.stream(type.getGenericsTypes()) .map(gt -> gt.getType()) @@ -551,9 +622,9 @@ private Statement processInputV1(ProcessInputContext ctx) { if( ctx.statement() != null ) { result = statement(ctx.statement()); } - else if( ctx.identifier().size() == 1 && ctx.type() == null ) { + else if( ctx.identifier() != null && ctx.type() == null ) { // identifier with no type annotation should be parsed as legacy input declaration - result = ast( stmt(variableName(ctx.identifier().get(0))), ctx ); + result = ast( stmt(variableName(ctx.identifier())), ctx ); } else { collectSyntaxError(new SyntaxException("Typed input declaration is not allowed in legacy process -- set `nextflow.preview.types = true` to use typed processes in this script", ast(new EmptyStatement(), ctx))); @@ -586,6 +657,9 @@ private Statement processOutputsV2(ProcessOutputsContext ctx) { collectSyntaxError(new SyntaxException("Every output must be assigned to a name when there are multiple outputs", result)); return null; } + if( !hasEmitExpression && statements.size() > 1 ) { + collectWarning("Typed process should have only one output -- consider combining outputs into a record", ctx.OUTPUT().getText(), ast( new EmptyStatement(), ctx.OUTPUT() )); + } return result; } @@ -870,7 +944,7 @@ private Statement workflowPublisher(WorkflowEmitContext ctx) { var target = nameTypePair(ctx.nameTypePair()); Statement result; if( ctx.expression() != null ) { - var source = expression(ctx.expression()); + var source = expression(ctx.expression()); result = stmt(ast( new AssignmentExpression(target, source), ctx )); } else { @@ -1929,7 +2003,7 @@ private ClassNode type(TypeContext ctx, boolean allowProxy) { result.setGenericsTypes( typeArguments(ctx.typeArguments()) ); if( ctx.QUESTION() != null ) result.putNodeMetaData(ASTNodeMarker.NULLABLE, Boolean.TRUE); - return result; + return ast( result, ctx ); } throw createParsingFailedException("Unrecognized type: " + ctx.getText(), ctx); diff --git a/modules/nf-lang/src/main/java/nextflow/script/types/Channel.java b/modules/nf-lang/src/main/java/nextflow/script/types/Channel.java index b691d8f69d..605de7dc51 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/types/Channel.java +++ b/modules/nf-lang/src/main/java/nextflow/script/types/Channel.java @@ -261,6 +261,7 @@ Channel join( [Read more](https://nextflow.io/docs/latest/reference/operator.html#mix) """) Channel mix(Channel... others); + Channel mix(Value... others); @Operator @Description(""" diff --git a/modules/nf-lang/src/main/java/nextflow/script/types/Types.java b/modules/nf-lang/src/main/java/nextflow/script/types/Types.java index 9ba77a1ba8..76169441e4 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/types/Types.java +++ b/modules/nf-lang/src/main/java/nextflow/script/types/Types.java @@ -50,6 +50,18 @@ public class Types { ClassHelper.makeCached(MemoryUnit.class) ); + /** + * Determine whether a type is a functional interface. + * + * @param type + */ + public static boolean isFunctionalInterface(ClassNode type) { + return type.getAnnotations().stream() + .filter(an -> an.getClassNode().getName().equals(FunctionalInterface.class.getName())) + .findFirst() + .isPresent(); + } + /** * Determine whether a class is a namespace. * @@ -71,7 +83,7 @@ public static String getName(ClassNode type) { if( type.isArray() ) return getName(type.getComponentType()); - if( ClassHelper.isFunctionalInterface(type) ) + if( isFunctionalInterface(type) ) return closureName(type); return typeName(type); diff --git a/modules/nf-lang/src/test/groovy/nextflow/script/formatter/ScriptFormatterTest.groovy b/modules/nf-lang/src/test/groovy/nextflow/script/formatter/ScriptFormatterTest.groovy index dbad332217..6efa6eaef3 100644 --- a/modules/nf-lang/src/test/groovy/nextflow/script/formatter/ScriptFormatterTest.groovy +++ b/modules/nf-lang/src/test/groovy/nextflow/script/formatter/ScriptFormatterTest.groovy @@ -231,6 +231,27 @@ class ScriptFormatterTest extends Specification { } ''' ) + + checkFormat( + '''\ + nextflow.preview.types=true + + process hello{ + input: (id:String,infile:Path):Record ; script: 'cat input.txt > output.txt' + } + ''', + '''\ + nextflow.preview.types = true + + process hello { + input: + (id: String, infile: Path): Record + + script: + 'cat input.txt > output.txt' + } + ''' + ) } def 'should format a function definition' () {