feat(wave): Support fetching local files in FUSION_CONTAINER_CONFIG_URL#7142
Merged
Conversation
✅ Deploy Preview for nextflow-docs-staging canceled.
|
…URL` This change allows `FUSION_CONTAINER_CONFIG_URL` to accept a local filesystem path (an absolute path or a `file://` URI) in addition to an http(s) URL. This lets engineers iterate on Fusion manifests without publishing them to a reachable URL on every change. Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
alberto-miranda
force-pushed
the
comp-869/local-fusion-manifest
branch
from
681c19b to
4583a3d
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The previous `url.startsWith('file:')` check admitted shapes that later
crash deep in `Paths.get(URI)` (e.g. `file://host/...`,
`file:relative.json`). Parse to URI and require `scheme == 'file'`,
empty/null authority, and an absolute path so misconfigured values fail
fast with a clear error.
Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
`fetchContainerConfig(Path)` surfaced raw NIO exceptions (`NoSuchFileException`, `AccessDeniedException`, etc.) with no indication that they came from the Fusion container config path. Wrap in `IllegalStateException` carrying the offending path so failures are attributable to the originating config value. Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
The `amd64`/`arm64` regex substitution applies to the entire URI string regardless of scheme, so a local manifest at `file:///path/fusion-amd64.json` is silently rewritten on `arm64` tasks. Make the helper a no-op for `file:` URIs — users pointing at a specific local file expect Wave to read exactly that file. Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
The `fetchContainerConfig(URI)` dispatcher is the only external entry point, and it is already `@Memoized` — repeat calls for the same URI never reach the `URL`/`Path` overloads. Their caches therefore hold entries that can never be hit. Keep memoization on the dispatcher only. Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
Note the supported URL schemes in both the reference docs and the `@Description` so users know `file:/...` is a valid value, not an undocumented side-effect of the validator. Signed-off-by: Alberto Miranda <alberto.miranda@seqera.io>
Collaborator
Author
|
@pditommaso I addressed your comments but the docs workflow seems to be failing because:
I'm happy to fix it in this PR or a follow up, but don't want to muck up the existing setup, so I would appreciate some guidance on the preferred approach 🙏 |
Member
|
Sorry for the long delay. All set here! |
…ast] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
pditommaso
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change allows
FUSION_CONTAINER_CONFIG_URL(and the corresponding Nextflow config optionfusion.containerConfigUrl) to accept a local filesystem path as an alternative to an http(s) URL. The path must be expressed as afile:URI.Note
This PR resurrects #5738, adapted to the current changes in
master.Rationale
Today,
FUSION_CONTAINER_CONFIG_URLonly acceptshttp(s)URLs. The Wave client fetches the JSON manifest from the provider URL and applies the declared layers to the task container. This works fine in production, but makes iterating on manifest changes painful: any change (e.g. bumping a Fusion layer, swapping in a development build, or testing a new Fusion Snapshots variant) means publishing a new JSON to a reachable URL before every run, then chasing down stale artefacts in object storage afterwards.By accepting a local manifest, it is possible to decouple the development loop from the production pipeline: engineers can edit a JSON on disk and re-run a pipeline immediately, instead of round-tripping every change through object storage and an externally reachable URL.
Changes
FusionConfig.containerConfigUrl()toFusionConfig.containerConfigURI()(return typeURL→URI) sofile:URIs travel cleanly through the resolution path.FusionConfig.validProtocol()to also acceptfile:URIs.FusionConfig.getConfig().WaveClient.resolveContainerConfig()now collectsList<URI>(previouslyList<URL>).replaceFusionArch()to acceptURI(no-op on local manifests, since the arch-suffix regex doesn't match).fetchContainerConfig(URI):http/httpsto the existing HTTP path,file:to a newfetchContainerConfig(Path)that reads from disk,FusionConfigTestandWaveClientTest.