Skip to content

Conversation

@kev-cao
Copy link
Contributor

@kev-cao kev-cao commented Dec 23, 2025

This commit adds the ListRestorableBackup helper, which reads through the index and returns all restorable times along with their associated backup IDs.

Epic: CRDB-57536

Informs: #159647

Release note: None

This commit updates the `ExternalStorage` `List` interface to take in an
options struct and moves `delimiter` into the struct. This is to
facilitate further work in adding more options to the `List` call.

Epic: None

Release note: None
This patch adds an AfterKey option in ExternalStorage.List. If it is
specified, only names that are lexicographically greater than AfterKey
are returned.

Epic: CRDB-57536

Informs: cockroachdb#159647

Release note: None
@kev-cao kev-cao requested review from a team as code owners December 23, 2025 01:43
@kev-cao kev-cao requested review from golgeek, srosenberg and xxmplus and removed request for a team December 23, 2025 01:43
@blathers-crl
Copy link

blathers-crl bot commented Dec 23, 2025

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@kev-cao kev-cao requested review from ZhouXing19, msbutler and rharding6373 and removed request for a team, ZhouXing19, golgeek, rharding6373, srosenberg and xxmplus December 23, 2025 01:43
@cockroach-teamcity
Copy link
Member

This change is Reviewable

This commit adds the `ListRestorableBackup` helper, which reads through
the index and returns all restorable times along with their associated
backup IDs.

Epic: CRDB-57536

Informs: cockroachdb#159647

Release note: None
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new ListRestorableBackups helper function that reads through the backup index to return all restorable backups within a specified time range, along with their backup IDs. To support this functionality, the PR refactors the ExternalStorage.List method signature across all cloud storage implementations to use a structured ListOptions parameter instead of individual delimiter and prefix parameters, adding support for AfterKey filtering.

  • Refactored all List method signatures to accept a ListOptions struct instead of separate delimiter and prefix parameters
  • Added ListRestorableBackups function that returns restorable backups with IDs within a time range, with logic to elide compacted duplicates
  • Implemented AfterKey filtering support in all cloud storage providers (S3, GCS, Azure, nodelocal, userfile) with client-side filtering for consistency

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pkg/cloud/external_storage.go Defines new ListOptions struct with Delimiter and AfterKey fields, along with CanonicalAfterKey helper method
pkg/backup/backupinfo/backup_index.go Adds ListRestorableBackups, listIndexesWithinRange, and related helper functions for parsing index paths and encoding backup IDs
pkg/backup/backupinfo/backup_index_test.go Adds comprehensive test coverage for ListRestorableBackups with various backup chain scenarios including compacted and revision-history backups
pkg/cloud/gcp/gcs_storage.go Updates List to use ListOptions and implements client-side AfterKey filtering
pkg/cloud/azure/azure_storage.go Updates List to use ListOptions and implements client-side AfterKey filtering for both blob prefixes and items
pkg/cloud/amazon/s3_storage.go Updates List to use ListOptions and implements client-side AfterKey filtering
pkg/cloud/nodelocal/nodelocal_storage.go Updates List to use ListOptions and implements client-side AfterKey filtering with delimiter grouping support
pkg/cloud/userfile/file_table_storage.go Updates List to use ListOptions and implements client-side AfterKey filtering with delimiter grouping support
pkg/cloud/httpsink/http_storage.go Updates List method signature to accept ListOptions (no-op implementation)
pkg/cloud/nullsink/nullsink_storage.go Updates List method signature to accept ListOptions (no-op implementation)
pkg/cloud/impl_registry.go Updates esWrapper.List to pass through ListOptions to wrapped storage
pkg/cloud/cloudtestutils/cloud_test_helpers.go Adds comprehensive test cases for AfterKey filtering behavior with various prefix and delimiter combinations
pkg/sql/importer/*.go Updates all List call sites to use cloud.ListOptions{}
pkg/sql/bulkutil/*.go Updates all List call sites to use cloud.ListOptions{}
pkg/backup/backupinfo/manifest_handling.go Updates List calls to use cloud.ListOptions{Delimiter: ...}
pkg/backup/backupdest/*.go Updates all List call sites to use cloud.ListOptions{} with appropriate delimiter settings
pkg/backup/backupencryption/encryption.go Updates List call to use cloud.ListOptions{Delimiter: ...}
pkg/backup/backup_job.go Updates List call to use cloud.ListOptions{}
pkg/backup/backup_test.go Updates all List call sites to use cloud.ListOptions{}
pkg/storage/shared_storage.go Updates List call to use cloud.ListOptions{Delimiter: delimiter}
pkg/roachprod/blobfixture/registry.go Updates all List call sites to use cloud.ListOptions{}
pkg/cmd/roachtest/tests/cdc_helper.go Updates List call to use cloud.ListOptions{}
pkg/ccl/workloadccl/fixture.go Updates List call to use cloud.ListOptions{Delimiter: "/"}
pkg/ccl/changefeedccl/sink_cloudstorage_test.go Updates mock List method signature to accept ListOptions
pkg/cli/userfile.go Updates all List call sites to use cloud.ListOptions{}
pkg/cloud/cloudtestutils/cloud_nemesis.go Updates List call to use cloud.ListOptions{}
pkg/cloud/externalconn/utils/connection_utils.go Updates List call to use cloud.ListOptions{}
pkg/backup/backupinfo/BUILD.bazel Adds dependency on //pkg/util/besteffort package

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants