Skip to content

Conversation

dglazer
Copy link

@dglazer dglazer commented Sep 26, 2025

… storage

  • Add logsBucket configuration option to BatchConfig
  • Support both CLOUD_LOGGING (default) and PATH log destinations
  • Add validation for GCS bucket paths (must start with gs://)
  • Include comprehensive tests for new functionality
  • Maintain backward compatibility

🤖 Generated with Claude Code
NOTE: this change still needs human review.

Signed-off-by: David Glazer [email protected]

Copy link

netlify bot commented Sep 26, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 9cba44c
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/68dfe3b2b8f90500085c8b6a

@dglazer dglazer marked this pull request as draft September 26, 2025 14:00
@emmarogge emmarogge force-pushed the fix-issue-6182 branch 3 times, most recently from be1236d to 71f6f4a Compare September 29, 2025 19:35
… bucket log storage

- Add logsBucket configuration option to BatchConfig
- Support both CLOUD_LOGGING (default) and PATH log destinations
- Add validation for GCS bucket paths (must start with gs://)
- Include comprehensive tests for new functionality
- Maintain backward compatibility

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: David Glazer <[email protected]>
David Glazer and others added 2 commits September 29, 2025 19:39
- Add helper methods to extract bucket names and convert GCS paths to mount paths
- Ensure logs bucket is mounted as Volume in GoogleBatchScriptLauncher
- Update LogsPolicy to use container mount paths instead of GCS paths
- Add comprehensive tests for bucket mounting and path conversion
- Addresses reviewer feedback about missing bucket mounting requirements

This ensures Google Batch can write logs to the specified GCS bucket by
properly mounting it before referencing it in the LogsPolicy PATH.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Emma Rogge <[email protected]>
1. Rename variable due to existing variable with same name.
2. Specify parameter type as string.

Signed-off-by: Emma Rogge <[email protected]>
@vdauwera
Copy link

vdauwera commented Oct 2, 2025

@bentsherman I had a quick chat with @dglazer, this PR is ready for review (sounds like marking the PR as not draft can only be done by someone with write access? idk)

@bentsherman bentsherman marked this pull request as ready for review October 2, 2025 17:50
@bentsherman
Copy link
Member

@dglazer thanks for the contribution. What's the benefit of being able to send logs to a bucket instead of the standard cloud logging?

@dglazer
Copy link
Author

dglazer commented Oct 2, 2025

Thanks for taking a look @bentsherman . The short answer is that it lets us provide a unified management experience to researchers, with the same ACLs controlling access to workflow inputs, outputs, and logs, and without them having to be familiar with GCP-native logging tools. @emmarogge can provide more details if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants