Skip to content

Conversation

pp-mo
Copy link
Member

@pp-mo pp-mo commented Sep 30, 2025

Closes #6374

This approach uses f3fs-FUSE

Tested: have shown that this can load netcdf and PP files from an S3 bucket.

Much to-do

  • add f3fs-FUSE to deps
  • lots of tests
  • consider saving
  • further explore the "unmount problems"

Copy link

codecov bot commented Sep 30, 2025

Codecov Report

❌ Patch coverage is 5.76923% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.93%. Comparing base (8ed2570) to head (ada62dd).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
lib/iris/io/s3_mounts.py 0.00% 92 Missing ⚠️
lib/iris/loading.py 50.00% 5 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6731      +/-   ##
==========================================
- Coverage   90.29%   89.93%   -0.36%     
==========================================
  Files          91       92       +1     
  Lines       24472    24577     +105     
  Branches     4569     4583      +14     
==========================================
+ Hits        22097    22104       +7     
- Misses       1607     1704      +97     
- Partials      768      769       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pp-mo
Copy link
Member Author

pp-mo commented Oct 1, 2025

Update: @ukmo-ccbunney pointed out that it could be simpler to leave "s3fs" mounting to the user.
That certainly looks like a simpler approach, and doesn't mire us in some complex issues which this approach raises:

  • testing is not too simple
  • introduces awkward dependencies
  • could be tricky to maintain
  • apparently, works for Linux/macOS/FreeBSD -- but not elsewhere
  • the indeterminate unmount problem ("umount target is busy") is a worry -- especially if it could have security implications

What we might do is provide some simple advice in the Iris docs under "Further Topics" ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Extend Iris loading to accept S3 URLs

1 participant