ADR: Nextflow Module System

[pditommaso]

Summary

Architecture Decision Record (ADR) for the Nextflow Module System, documenting:

• Remote module inclusion via registry with @scope/name syntax
• Semantic versioning with dependency resolution
• Unified Nextflow Registry (rebrand existing plugin registry)
• First-class CLI support (pull, push, search, run)
• Module metadata schema (meta.yaml) with JSON Schema validation
• Structured tool arguments replacing ext.args pattern

Key Files

• adr/20251114-module-system.md - Main ADR document with full specification
• adr/module-spec-schema.json - JSON Schema for meta.yaml validation

Updates

Version 2.2 (2025-01-06)

• Structured tool arguments: Added args property to tools section for type-safe argument configuration
• New implicit variables: tools.<toolname>.args.<argname> returns formatted flag+value; tools.<toolname>.args returns all args concatenated
• Deprecation: ext.args, ext.args2, ext.args3 pattern deprecated in favor of structured tool arguments

Version 2.1 (2025-12-11)

• Unified dependencies: Consolidated components, dependencies, and requires into single requires field
• New sub-properties: requires.modules and requires.workflows for declaring module dependencies
• Unified version syntax: [scope/]name[@constraint] format across plugins, modules, and workflows
• Deprecation: components field deprecated (use requires.modules instead)

Test plan

• Review ADR document for completeness and clarity
• Validate JSON Schema against example meta.yaml files
• Review tool arguments specification for nf-core compatibility
• Validate design decisions align with ecosystem needs

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	68b202f
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69690cda0b4b1800098c088a

[bentsherman]

The module spec schema will be defined in the nextflow-io/schemas repo. Here is the PR for it: nextflow-io/schemas#10

[bentsherman]

This should be out-of-scope for the first iteration. Let's focus on the modules command and module spec generation

[jorgee]

If I have understood correctly, at parse time @nf-core/bwa-align will be translated to modules/@scope/name@version/main.nf.
If it is not supported in first iteration, users must write the include statements providing the module path taking into account the version and keep the coherence of these include statements with the versions of nextflow.config.
I think it is also required for defining modules importing other modules. If this is not supported, modules must be written using include with paths.

[bentsherman]

Yes, likely module versioning would also be out of scope. The primary motivation initially was to allow remote module execution:

nextflow module run @nf-core/bwa-align

This can be done without remote module inclusion. We can probably still implement module versioning just for installing and executing modules

[pditommaso]

There's no need to delay module versioning, there's already an initial implementation for the registry backend

[jorgee]

There could be cases where the version in cached modules does not match the nextflow.config (e.g users upgrade the module in config but do not commit the module's files)
What should it do in these cases? warn the user and download the new version or fail?

[pditommaso]

Likely it should be added a command to detect/solve these kind of inconsistency but I would avoid warning too verbosely

[jorgee]

Should it also include the installed transitive dependencies in nexflow.config, or just the module requested to install.

[jorgee]

Modules meta includes required Nextflow version and plugins. At this point, I think it should validate the required nextflow version and the required plugins (loading them if missing)

[pditommaso]

Summary of Recent Changes

Major Simplification: Module Storage and Verification Model

The module system has been significantly simplified by making the registry the source of truth for checksums:

Config Syntax Simplified - Removed checksums from config, version-only format:

modules {
    '@nf-core/salmon' = '1.1.0'
    '@nf-core/bwa-align' = '1.2.0'
}

Local Storage Structure - Single version per module (no version in path), with .checksum file for integrity verification:

modules/
└── @nf-core/
    └── bwa-align/
        ├── .checksum           # Cached registry checksum for fast validation
        ├── meta.yaml
        └── main.nf

Verification Flow:

1. Install: Download module → store .checksum from registry's X-Checksum header
2. On run: Compute local checksum → compare with .checksum
3. Match → proceed (no registry call needed)
4. Mismatch → report warning (user must explicitly reinstall)

freeze Command Simplified - Only pins versions, no checksums in config

Other Updates

• Module API updated to match actual registry implementation (/api/modules/{name} pattern)
• Tool Arguments CLI support added: --tools:<tool>:<arg> <value> syntax
• Open Questions section added for pending design decisions
• Terminology standardized: "Nextflow registry" instead of "plugin registry"
• Various fixes: consistent sha256: prefix, license spelling, authors plural form, path formats

[pditommaso]

Version 2.4 Changes

Updated the ADR to remove transitive dependency resolution from the initial implementation scope. This simplifies the module system by requiring explicit dependency declarations.

Summary of Changes

• Removed freeze command from CLI - no longer needed without transitive dependency management
• Simplified install behavior - removed automatic recursive installation of transitive dependencies
• Updated remove command - removed orphaned transitive dependency cleanup
• Simplified dependency resolution flow - modules are resolved directly from nextflow.config declarations only
• Updated rationale and consequences sections to reflect the simplified model

Rationale

Transitive dependency resolution adds significant complexity and will not be implemented in this initial stage. Each module's dependencies must be explicitly declared in nextflow.config, giving users full control and visibility over what modules are installed.

This change also affects the spec at specs/251117-module-system/spec.md which was updated to remove:

• Transitive dependency acceptance scenarios
• FR-010, FR-011 (transitive dependency requirements)
• freeze command from FR-019
• SC-008 (transitive dependencies success criteria)