Add heap-based LIMIT_SORT driver and migrate to ioredis - #427
Open
NuhYlmz wants to merge 3 commits into
Open
Conversation
…roved compatibility. Added new lodash dependencies and removed obsolete references to _redisOplogCollectionMethods in ObservableCollection and RedisSubscriptionManager.
Use MaxHeap/MinMaxHeap for published and unpublished buffers to avoid full requeries on eligible inserts/updates/removes, require Meteor 3.4 and binary-heap, and fall back to requery when skip is present or heaps diverge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
redisnpm package withioredis5.4.1REDIS_OPLOG_URLenvironment variable supportOplogObserveDriverbinary-heappackageProblem
Publications with
limit + sortuse theLIMIT_SORTstrategy. Previously, almost every insert, update, or remove triggered a fullrequery()— a sorted, limited MongoDBfetchAsyncon every Redis event, on top of thefindOneAsyncalready needed to load the changed document. Under high write volume, if that sorted fetch is slow (large collection, missing index, complex selector), events are processed sequentially and each one blocks until the requery completes. This causes a growing backlog and subscriptions falling behind real-time changes.Solution
This implementation is based on Meteor's native oplog tailing driver (
OplogObserveDriver), which uses the same heap-based approach forlimit + sortqueries. The goal is behavioral parity with Meteor's built-in oplog implementation, not a custom optimization.For
limit + sortqueries (withoutskip), we maintain two in-memory heaps matching Meteor's design:_published(MaxHeap) — the top-N documents currently sent to clients_unpublishedBuffer(MinMaxHeap) — the next N candidatesThe heap lifecycle (
_addPublished,_removePublished,_addMatching,_removeMatching,_handleDocChange) and the fallback requery path (_fallbackRequeryLimitSort) mirror the corresponding methods in Meteor'sOplogObserveDriver.Instead of re-running a full sorted fetch on every change, a single
findOneAsyncfor the changed document (already performed when the Redis event is received) is sufficient for most events. The heap uses that document together with the in-memory published set and buffer to decide whether to publish, buffer, promote, or evict — all via O(log n) comparator operations, with no additional MongoDB round-trip. A full fetch is only needed on initial subscription setup and in rare fallback cases when buffer state becomes uncertain (_fallbackRequeryLimitSort, fetching2 × limitprojected documents).Breaking changes
retryIntervalMssetting removed from config and README