Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add backward seek-based pagination support for the search endpoint #10793

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

eth3lbert
Copy link
Contributor

Previously, we were returning null for prev_page in search endpoint. With this PR, we now provide the URL for prev_page. For other endpoints to support backward seeking, they must have the page options with seek backward enabled (.enable_seek_backward(true)). Otherwise, without this, any seek param starting with - (indicating backward seeking) will result in an error!

It would be easier to review this PR commit by commit 😉

Resolves #10349.

Verified

This commit was signed with the committer’s verified signature.
…de()`

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
This makes "seek=" and "seek=-" valid, and "seek=-" could be treated as
paginating backward from the tail.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
...based on direction
Most of this should be intuitive, as it simply flips the comparison
operator. To make it easier to identify the difference, I chose to
implement it in a single function and place it next to the forward
ordering conditions. The only part that requires a bit of thought is the
case involving nulls last.

Verified

This commit was signed with the committer’s verified signature.
…` fn

Verified

This commit was signed with the committer’s verified signature.
…` fn

This also remove the obsolete `Paginated::is_explicit_page()`.

Verified

This commit was signed with the committer’s verified signature.
…d pagination

...when seeking backward involved.

Verified

This commit was signed with the committer’s verified signature.
…e to page

For instance, given rows with the following:

| a | b | c | ... | 1 | 2 | 3 |

When performing forward seek pagination with page size as 3 to the last
page, let's say we have the last page as 1, 2, 3. When performing
backward seek pagination, we reverse the order of the query so that we
can retrieve the data from the tail. This results in 3, 2, 1. However,
we would like to maintain consistent ordering as we view it with forward
pagination, namely 1, 2, 3. Therefore, we need to reverse the results
again to maintain the original ordering.

Verified

This commit was signed with the committer’s verified signature.
…agination

Verified

This commit was signed with the committer’s verified signature.
@eth3lbert eth3lbert added C-enhancement ✨ Category: Adding new behavior or a change to the way an existing feature works A-backend ⚙️ labels Mar 10, 2025

Verified

This commit was signed with the committer’s verified signature.
@eth3lbert eth3lbert force-pushed the seek-pagination-backward branch from a05bcbf to 44d5e67 Compare March 10, 2025 09:55
@eth3lbert eth3lbert requested review from Turbo87 and LawnGnome and removed request for Turbo87 March 10, 2025 10:13
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently only tested with a page size of 1. It would be better to expand the tests to include varying page sizes.

@eth3lbert eth3lbert requested a review from Turbo87 March 10, 2025 10:20
Copy link
Member

@Turbo87 Turbo87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still wondering if all the additional complexity is really worth it... 😅

I'm curious what the rest of the @rust-lang/crates-io team thinks about this.

@LawnGnome
Copy link
Contributor

Sorry, I skimmed this yesterday, but didn't have time to give it a full review. I still haven't, but my rough thought process was basically "oh, this is way better than I would have thought", then I realised that GitHub had collapsed the krate/search.rs changes, and I didn't feel that way so much after that.

That said, the changes are mostly mechanical, and feel like we could eventually abstract at least some of them behind a macro or two. Against that, krate/search.rs is already terrifying, and adding two new sets of branches for each sort option doesn't spark joy.

So, I guess I wouldn't block this, but I'm not super enthusiastic about it, either. Do we have a strong use case for this? (Maybe more a question for @GuillaumeGomez.) I'm probably -0 right now, but could be convinced, particularly if we came up with a way to abstract some of the details out of the controllers proper.

@eth3lbert
Copy link
Contributor Author

eth3lbert commented Mar 11, 2025

Thanks for the feedback!

Sorry, I skimmed this yesterday, but didn't have time to give it a full review. I still haven't, but my rough thought process was basically "oh, this is way better than I would have thought", then I realised that GitHub had collapsed the krate/search.rs changes, and I didn't feel that way so much after that.

That said, the changes are mostly mechanical, and feel like we could eventually abstract at least some of them behind a macro or two. Against that, krate/search.rs is already terrifying, and adding two new sets of branches for each sort option doesn't spark joy.

Cursor-based pagination does indeed complicate things, especially with NULLS LAST ordering and ordering with a "tie-breaker". Abstraction should be doable. Given the columns and their ordering, we should be able to build the filter query. However, that would require more work and potentially become more complex when more tables or even aliases are involved, and it could also be more difficult to implement with diesel. Ultimately, this would result in something similar to SeaORM's general cursor-based pagination solution, which would be great if we contribute to upstream 🤔.

So, I guess I wouldn't block this, but I'm not super enthusiastic about it, either. Do we have a strong use case for this? (Maybe more a question for @GuillaumeGomez.) I'm probably -0 right now, but could be convinced, particularly if we came up with a way to abstract some of the details out of the controllers proper.

This currently serves mainly for docs.rs, and I'm not personally pressing for it :D

@GuillaumeGomez
Copy link
Member

Well, the navigation experience is suboptimal, so that would be very welcome. However if you think that a better (technical) solution could be achieved, then we can wait for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-backend ⚙️ C-enhancement ✨ Category: Adding new behavior or a change to the way an existing feature works
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prev_page field is always null on search
4 participants