Generate plugin spec (#6361)

---------

Signed-off-by: Ben Sherman <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>

Author: bentsherman

CLAUDE.md

@@ -104,6 +104,7 @@ The project follows a modular architecture with a plugin-based system for cloud
 - `build.gradle`: Root build configuration with multi-module setup
 - `settings.gradle`: Gradle project structure definition
 - `plugins/*/VERSION`: Define the version of the corresponding plugin sub-project.
+- `adr/`: Architecture Decision Records (ADRs) documenting significant structural and technical decisions in the project
 
 ## Release process
 

adr/20250922-plugin-spec.md

@@ -0,0 +1,154 @@
+# Plugin Spec
+
+- Authors: Ben Sherman
+- Status: accepted
+- Deciders: Ben Sherman, Paolo Di Tommaso
+- Date: 2025-09-22
+- Tags: plugins
+
+## Summary 
+
+Provide a way for external systems to understand key information about third-party plugins.
+
+## Problem Statement
+
+Nextflow plugins need a way to statically declare extensions to the Nextflow language so that external systems can extract information about a plugin without loading it in the JVM.
+
+Primary use cases:
+
+- The Nextflow language server needs to know about any config scopes, custom functions, etc, defined by a plugin, in order to recognize them in Nextflow scripts and config files.
+
+- The Nextflow plugin registry (or other user interfaces) can use this information to provide API documentation.
+
+## Goals or Decision Drivers
+
+- External systems (e.g. language server) need to be able to understand plugins without having to load them in the JVM.
+
+## Non-goals
+
+- Defining specs for the core runtime and core plugins: these definitions are handled separately, although they may share some functionality with plugin specs.
+
+## Considered Options
+
+### Nextflow plugin system
+
+Require external systems to use Nextflow's plugin system to load plugins at runtime in order to extract information about them.
+
+- **Pro:** Allows any information to be extracted since the entire plugin is loaded
+
+- **Con:** Requires the entire Nextflow plugin system to be reused or reimplemented. Not ideal for Java applications since the plugin system is implemented in Groovy, incompatible with non-JVM applications
+
+- **Con:** Requires plugins to be downloaded, cached, loaded in the JVM, even though there is no need to use the plugin.
+
+### Plugin spec
+
+Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON.
+
+- **Pro:** Allows any system to inspect plugin definitions through a standard JSON document, instead of downloading plugins and loading them into a JVM.
+
+- **Con:** Requires the plugin spec to be generated at build-time and stored in the plugin registry.
+
+- **Con:** Requires a standard format to ensure interoperability across different versions of Nextflow, the language server, and third-party plugins.
+
+## Solution
+
+Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON.
+
+- Plugin developers only need to define [extension points](https://nextflow.io/docs/latest/plugins/developing-plugins.html#extension-points) as usual, and the Gradle plugin will extract the plugin spec and store it in the plugin registry as part of each plugin release.
+
+- The language server can infer which third-party plugins are required from the `plugins` block in a config file. It will retrieve the appropriate plugin specs from the plugin registry.
+
+A plugin spec consists of a list of *definitions*. Each definition has a *type* and a *spec*.
+
+Example:
+
+```json
+{
+    "$schema": "https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json",
+    "definitions": [
+        {
+            "type": "ConfigScope",
+            "spec": {
+                // ...
+            }
+        },
+        {
+            "type": "Function",
+            "spec": {
+                // ...
+            }
+        },
+    ]
+}
+```
+
+The following types of definitions are allowed:
+
+**ConfigScope**
+
+Defines a top-level config scope. The spec consists of a *name*, an optional *description*, and *children*.
+
+The children should be a list of definitions corresponding to nested config scopes and options. The following definitions are allowed:
+
+- **ConfigOption**: Defines a config option. The spec consists of a *description* and *type*.
+
+- **ConfigScope**: Defines a nested config scope, using the same spec as for top-level scopes.
+
+Example:
+
+```json
+{
+    "type": "ConfigScope",
+    "spec": {
+        "name": "hello",
+        "description": "The `hello` scope controls the behavior of the `nf-hello` plugin.",
+        "children": [
+            {
+                "type": "ConfigOption",
+                "spec": {
+                    "name": "message",
+                    "description": "Message to print to standard output when the plugin is enabled.",
+                    "type": "String"
+                }
+            }
+        ]
+    }
+}
+```
+
+**Factory**
+
+Defines a channel factory that can be included in Nextflow scripts. The spec is the same as for functions.
+
+**Function**
+
+Defines a function that can be included in Nextflow scripts. The spec consists of a *name*, an optional *description*, a *return type*, and a list of *parameters*. Each parameter consists of a *name* and a *type*.
+
+Example:
+
+```json
+{
+    "type": "Function",
+    "spec": {
+        "name": "sayHello",
+        "description": "Say hello to the given target",
+        "returnType": "void",
+        "parameters": [
+            {
+                "name": "target",
+                "type": "String"
+            }
+        ]
+    }
+}
+```
+
+**Operator**
+
+Defines a channel operator that can be included in Nextflow scripts. The spec is the same as for functions.
+
+## Rationale & discussion
+
+Now that there is a Gradle plugin for building Nextflow plugins and a registry to publish and retrieve plugins, it is possible to generate, publish, and retrieve plugin specs in a way that is transparent to plugin developers.
+
+Plugins specs adhere to a pre-defined [schema](https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json) to ensure consistency across different versions of Nextflow. In the future, new versions of the schema can be defined as needed to support new behaviors or requirements.

adr/YYYYMMDD-template-name.md

@@ -0,0 +1,76 @@
+# [short title of solved problem and solution]
+
+- Authors: [who wrote the ADR]
+- Status: [draft | proposed | rejected | accepted | deprecated | … | superseded by [xxx](xxx.md)]
+- Deciders: [list everyone involved in the decision]  <!-- optional - to be formalised -->
+- Date: [YYYY-MM-DD when the decision was last updated]
+- Tags: [space and/or comma separated list of tags]
+
+Technical Story: [description | ticket/issue URL] <!-- optional -->
+
+## Summary 
+
+Quick description of the problem and the context. Should not take more than 2-3 lines.
+
+## Problem Statement
+
+Description of the technical problem to solve or to decision to make. This should be concise but provide all required details and the context related to the technical decision to be taken.
+
+## Goals or Decision Drivers
+
+Depending the context define clearly what are the goals or what are the most important decision drivers.
+
+- [driver 1, e.g., a force, facing concern, …]
+- [driver 2, e.g., a force, facing concern, …]
+- … <!-- numbers of drivers can vary -->
+
+## Non-goals
+
+Define what's out of the scope of this ADR.
+
+## Considered Options <!-- optional -->
+
+- [option 1]
+- [option 2]
+- [option 3]
+- … <!-- numbers of options can vary -->
+
+
+## Pros and Cons of the Options <!-- optional -->
+
+### [option 1]
+
+[example | description | pointer to more information | …] <!-- optional -->
+
+- Good, because [argument a]
+- Good, because [argument b]
+- Bad, because [argument c]
+- … <!-- numbers of pros and cons can vary -->
+
+### [option 2]
+
+[example | description | pointer to more information | …] <!-- optional -->
+
+- Good, because [argument a]
+- Good, because [argument b]
+- Bad, because [argument c]
+- … <!-- numbers of pros and cons can vary -->
+
+
+## Solution or decision outcome
+
+Summarize the solution or decision outcome in one-two lines.
+
+## Rationale & discussion
+
+Describe the solution or the decision outcome discussing how decision drivers have been applied and how it matches the declared goals. This section is expected to be concise though providing comprehensive description of the technical solution and covering all uncertainty or ambiguous points.
+
+## Links <!-- optional -->
+
+- [Link type](link to adr) <!-- example: Refined by [xxx](yyyymmdd-xxx.md) -->
+- … <!-- numbers of links can vary -->
+
+## More information
+
+- [What is an ADR and why should you use them](https://github.com/thomvaill/log4brains/tree/master#-what-is-an-adr-and-why-should-you-use-them)
+- [ADR GitHub organization](https://adr.github.io/)

docs/developer/diagrams/nextflow.plugin.mmd

@@ -6,9 +6,9 @@ classDiagram
 
     Plugins --> PluginsFacade : init
 
-    PluginsFacade "1" --> "*" PluginSpec : load
+    PluginsFacade "1" --> "*" PluginRef : load
 
-    class PluginSpec {
+    class PluginRef {
         id : String
         version : String
     }

modules/nextflow/src/main/groovy/nextflow/config/schema/JsonRenderer.groovy renamed to modules/nextflow/src/main/groovy/nextflow/plugin/spec/ConfigSpec.groovy

@@ -13,74 +13,58 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package nextflow.config.schema
+package nextflow.plugin.spec
 
-import groovy.json.JsonOutput
-import groovy.transform.TypeChecked
-import nextflow.plugin.Plugins
-import nextflow.script.dsl.Description
+import groovy.transform.CompileStatic
+import nextflow.config.schema.SchemaNode
 import nextflow.script.types.Types
 import org.codehaus.groovy.ast.ClassNode
 
-@TypeChecked
-class JsonRenderer {
-
-    String render() {
-        final schema = getSchema()
-        return JsonOutput.toJson(schema)
-    }
+/**
+ * Generate specs for config scopes.
+ *
+ * @author Ben Sherman <[email protected]>
+ */
+@CompileStatic
+class ConfigSpec {
 
-    private static Map<String,?> getSchema() {
-        final result = new HashMap<String,?>()
-        for( final scope : Plugins.getExtensions(ConfigScope) ) {
-            final clazz = scope.getClass()
-            final scopeName = clazz.getAnnotation(ScopeName)?.value()
-            final description = clazz.getAnnotation(Description)?.value()
-            if( scopeName == '' ) {
-                SchemaNode.Scope.of(clazz, '').children().each { name, node ->
-                    result.put(name, fromNode(node))
-                }
-                continue
-            }
-            if( !scopeName )
-                continue
-            final node = SchemaNode.Scope.of(clazz, description)
-            result.put(scopeName, fromNode(node, scopeName))
-        }
-        return result
+    static Map<String,?> of(SchemaNode node, String name) {
+        return fromNode(node, name)
     }
 
-    private static Map<String,?> fromNode(SchemaNode node, String name=null) {
+    private static Map<String,?> fromNode(SchemaNode node, String name) {
         if( node instanceof SchemaNode.Option )
-            return fromOption(node)
+            return fromOption(node, name)
         if( node instanceof SchemaNode.Placeholder )
-            return fromPlaceholder(node)
+            return fromPlaceholder(node, name)
         if( node instanceof SchemaNode.Scope )
             return fromScope(node, name)
         throw new IllegalStateException()
     }
 
-    private static Map<String,?> fromOption(SchemaNode.Option node) {
+    private static Map<String,?> fromOption(SchemaNode.Option node, String name) {
         final description = node.description().stripIndent(true).trim()
         final type = fromType(new ClassNode(node.type()))
 
         return [
-            type: 'Option',
+            type: 'ConfigOption',
             spec: [
+                name: name,
                 description: description,
                 type: type
             ]
         ]
     }
 
-    private static Map<String,?> fromPlaceholder(SchemaNode.Placeholder node) {
+    private static Map<String,?> fromPlaceholder(SchemaNode.Placeholder node, String name) {
         final description = node.description().stripIndent(true).trim()
         final placeholderName = node.placeholderName()
         final scope = fromScope(node.scope())
 
         return [
-            type: 'Placeholder',
+            type: 'ConfigPlaceholderScope',
             spec: [
+                name: name,
                 description: description,
                 placeholderName: placeholderName,
                 scope: scope.spec
@@ -90,34 +74,28 @@ class JsonRenderer {
 
     private static Map<String,?> fromScope(SchemaNode.Scope node, String scopeName=null) {
         final description = node.description().stripIndent(true).trim()
-        final children = node.children().collectEntries { name, child ->
-            Map.entry(name, fromNode(child, name))
+        final children = node.children().collect { name, child ->
+            fromNode(child, name)
         }
 
         return [
-            type: 'Scope',
+            type: 'ConfigScope',
             spec: [
-                description: withLink(scopeName, description),
+                name: scopeName,
+                description: description,
                 children: children
             ]
         ]
     }
 
-    private static String withLink(String scopeName, String description) {
-        return scopeName
-            ? "$description\n\n[Read more](https://nextflow.io/docs/latest/reference/config.html#$scopeName)\n"
-            : description
-    }
-
     private static Object fromType(ClassNode cn) {
         final name = Types.getName(cn.getTypeClass())
-        if( cn.isUsingGenerics() ) {
+        if( !cn.isGenericsPlaceHolder() && cn.getGenericsTypes() != null ) {
             final typeArguments = cn.getGenericsTypes().collect { gt -> fromType(gt.getType()) }
             return [ name: name, typeArguments: typeArguments ]
         }
         else {
             return name
         }
     }
-
 }