-
Notifications
You must be signed in to change notification settings - Fork 4k
backup: add ListRestorableBackup helper #160047
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
backup: add ListRestorableBackup helper #160047
Conversation
This commit updates the `ExternalStorage` `List` interface to take in an options struct and moves `delimiter` into the struct. This is to facilitate further work in adding more options to the `List` call. Epic: None Release note: None
This patch adds an AfterKey option in ExternalStorage.List. If it is specified, only names that are lexicographically greater than AfterKey are returned. Epic: CRDB-57536 Informs: cockroachdb#159647 Release note: None
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
This commit adds the `ListRestorableBackup` helper, which reads through the index and returns all restorable times along with their associated backup IDs. Epic: CRDB-57536 Informs: cockroachdb#159647 Release note: None
0aebfd4 to
385422a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR introduces a new ListRestorableBackups helper function that reads through the backup index to return all restorable backups within a specified time range, along with their backup IDs. To support this functionality, the PR refactors the ExternalStorage.List method signature across all cloud storage implementations to use a structured ListOptions parameter instead of individual delimiter and prefix parameters, adding support for AfterKey filtering.
- Refactored all
Listmethod signatures to accept aListOptionsstruct instead of separatedelimiterandprefixparameters - Added
ListRestorableBackupsfunction that returns restorable backups with IDs within a time range, with logic to elide compacted duplicates - Implemented
AfterKeyfiltering support in all cloud storage providers (S3, GCS, Azure, nodelocal, userfile) with client-side filtering for consistency
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/cloud/external_storage.go | Defines new ListOptions struct with Delimiter and AfterKey fields, along with CanonicalAfterKey helper method |
| pkg/backup/backupinfo/backup_index.go | Adds ListRestorableBackups, listIndexesWithinRange, and related helper functions for parsing index paths and encoding backup IDs |
| pkg/backup/backupinfo/backup_index_test.go | Adds comprehensive test coverage for ListRestorableBackups with various backup chain scenarios including compacted and revision-history backups |
| pkg/cloud/gcp/gcs_storage.go | Updates List to use ListOptions and implements client-side AfterKey filtering |
| pkg/cloud/azure/azure_storage.go | Updates List to use ListOptions and implements client-side AfterKey filtering for both blob prefixes and items |
| pkg/cloud/amazon/s3_storage.go | Updates List to use ListOptions and implements client-side AfterKey filtering |
| pkg/cloud/nodelocal/nodelocal_storage.go | Updates List to use ListOptions and implements client-side AfterKey filtering with delimiter grouping support |
| pkg/cloud/userfile/file_table_storage.go | Updates List to use ListOptions and implements client-side AfterKey filtering with delimiter grouping support |
| pkg/cloud/httpsink/http_storage.go | Updates List method signature to accept ListOptions (no-op implementation) |
| pkg/cloud/nullsink/nullsink_storage.go | Updates List method signature to accept ListOptions (no-op implementation) |
| pkg/cloud/impl_registry.go | Updates esWrapper.List to pass through ListOptions to wrapped storage |
| pkg/cloud/cloudtestutils/cloud_test_helpers.go | Adds comprehensive test cases for AfterKey filtering behavior with various prefix and delimiter combinations |
| pkg/sql/importer/*.go | Updates all List call sites to use cloud.ListOptions{} |
| pkg/sql/bulkutil/*.go | Updates all List call sites to use cloud.ListOptions{} |
| pkg/backup/backupinfo/manifest_handling.go | Updates List calls to use cloud.ListOptions{Delimiter: ...} |
| pkg/backup/backupdest/*.go | Updates all List call sites to use cloud.ListOptions{} with appropriate delimiter settings |
| pkg/backup/backupencryption/encryption.go | Updates List call to use cloud.ListOptions{Delimiter: ...} |
| pkg/backup/backup_job.go | Updates List call to use cloud.ListOptions{} |
| pkg/backup/backup_test.go | Updates all List call sites to use cloud.ListOptions{} |
| pkg/storage/shared_storage.go | Updates List call to use cloud.ListOptions{Delimiter: delimiter} |
| pkg/roachprod/blobfixture/registry.go | Updates all List call sites to use cloud.ListOptions{} |
| pkg/cmd/roachtest/tests/cdc_helper.go | Updates List call to use cloud.ListOptions{} |
| pkg/ccl/workloadccl/fixture.go | Updates List call to use cloud.ListOptions{Delimiter: "/"} |
| pkg/ccl/changefeedccl/sink_cloudstorage_test.go | Updates mock List method signature to accept ListOptions |
| pkg/cli/userfile.go | Updates all List call sites to use cloud.ListOptions{} |
| pkg/cloud/cloudtestutils/cloud_nemesis.go | Updates List call to use cloud.ListOptions{} |
| pkg/cloud/externalconn/utils/connection_utils.go | Updates List call to use cloud.ListOptions{} |
| pkg/backup/backupinfo/BUILD.bazel | Adds dependency on //pkg/util/besteffort package |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This commit adds the
ListRestorableBackuphelper, which reads through the index and returns all restorable times along with their associated backup IDs.Epic: CRDB-57536
Informs: #159647
Release note: None