Add Filesystem datastore support #5892
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new FilesystemDataStore implementation that enables storing DataStore data on the local filesystem, allowing quickstart to publish a meta lake locally without requiring cloud storage (GCS/S3). This unblocks galexie meta lake support in quickstart and integration testing scenarios.
Key Changes
- Implements all DataStore interface methods (Exists, Size, GetFile, PutFile, PutFileIfNotExists, GetFileMetadata, GetFileLastModified, ListFilePaths, Close) for filesystem-based storage
- Supports optional metadata sidecar files (.metadata.json) with configurable write_metadata flag
- Integrates FilesystemDataStore into the DataStore factory pattern
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 11 comments.
| File | Description |
|---|---|
| support/datastore/filesystem.go | Core implementation of FilesystemDataStore with file I/O operations, metadata handling, and directory traversal for listing files |
| support/datastore/filesystem_test.go | Comprehensive test suite covering all DataStore operations, metadata handling, edge cases, and configuration options |
| support/datastore/datastore.go | Registers "Filesystem" type in the DataStore factory function |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Draft
19 tasks
leighmcculloch
changed the title
sreuland
reviewed
Jan 7, 2026
sreuland
approved these changes
Jan 7, 2026
Contributor
sreuland
left a comment
sreuland
left a comment
There was a problem hiding this comment.
lgtm, great docs and test coverage also!
tamirms
approved these changes
Jan 7, 2026
Co-authored-by: shawn <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add a new FilesystemDataStore implementation that satisfies the DataStore interface. Store data on the local filesystem with configurable destination_path and optional metadata sidecar files (.metadata.json). Support all DataStore operations: Exists, Size, GetFile, PutFile, PutFileIfNotExists, GetFileMetadata, GetFileLastModified, ListFilePaths, and Close.
Why
Enable quickstart to publish a meta lake locally without requiring cloud storage configuration (GCS/S3) to support integration testing with galexie with quickstart.
Related:
Close #5891
Close stellar/stellar-galexie#10
Close stellar/quickstart#835