Make iroh-blobs fs storage agnostic

[seanaye]

I've been looking at both the fs store as well as the s3 store here.

It appears like the fs storage can be adapted to use any storage backend (e.g. s3) with some trait interface that abstracts away any std::fs.

Would you accept a PR for this? It would introduce breaking changes to the API but I think it would make supporting any storage mechanism much easier in the future. Eventually I think it would even be possible to support browser based storage like local storage or indexeddb which would allow the iroh blobs protocol to function in the browser.

[rklaehn]

We would happily accept a PR. But we are currently doing a major rework of blobs. So a lot of things will change soon.

We want to release iroh 1.0 in Q3 this year, and iroh-blobs 1.0 at the same time.

What was the change you had in mind? I would have to check how it maps to the new way to abstract over stores.

[seanaye]

Basically removing any direct references to any std::fs inside of the fs store directly. Then the caller could provide its own storage interface that implements a trait. By default we could pass the impl for std::fs if the caller doesn't provide one.

Is there a dev branch I could work off? If not I might just submit the MR. I don't think the diff would be overly large

[rklaehn]

So you want to use the fs store implementation, but with a different storage backend than files? Is that storage backend synchronous or asynchronous?

The current storage trait is for general abstraction over stores, but implementing a store can be quite a bit of work. We have rarely had the case where people wanted to put in the effort to write their own stores.

I am working on what will become blobs 1.0 in https://github.com/n0-computer/blobs2 , but it is still very much in flux. I will release a pre-alpha in the next two weeks and use it at a workshop at https://jonthebeach.com/, then in the next weeks it will be stabilised and will eventually replace the current blobs impl.

The blobs protocol as well as the redb/files based storage format for the fs store will remain unchanged, but the way to abstract over stores changes significantly.

[seanaye]

Yes that is correct: use the fs store implementation but allow you to bring your own backend (not necessarily filesystem). I opened a draft PR #86 so you can see the approach in a bit more detail. The backend is asynchronous (its easy to make sync code return a future but not so easy to go the other way).

I still have some work left to do to clean up that PR