Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Transparently fetch multiple documents grouped per block #2276

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

adamreichold
Copy link
Collaborator

@adamreichold adamreichold commented Dec 11, 2023

My personal take on the discussion in #2252, i.e. expose enough information and control to schedule concurrent reads into independent block caches as to maximize cache effectiveness and then expose this as Searcher::docs_async which yields an iterator of futures which user code can then poll concurrently or in parallel as it sees fit.

@adamreichold adamreichold changed the title RFC: Give user code insight into and control over block cache usage RFC: Transparently fetch multiple documents grouped per block Dec 11, 2023
&self,
doc_addresses: impl IntoIterator<Item = DocAddress>,
) -> crate::Result<
impl Iterator<Item = impl Future<Output = crate::Result<Vec<(DocAddress, D)>>>> + '_,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signature is admittedly somewhat unwieldy even though relatively easy to use in practice as the test shows. I wonder if I should add a no_run example showing usage with FuturesUnordered? (no_run to avoid making the example unreadable due to index setup code. Alternatively the setup code could be hidden.)

Copy link
Collaborator Author

@adamreichold adamreichold Dec 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a no_run example on how to use this with FuturesUnordered and block_on.

@adamreichold adamreichold force-pushed the block-cache-control branch 3 times, most recently from d00dd5a to a2953d6 Compare December 11, 2023 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant