Skip to content

Add heap-based LIMIT_SORT driver and migrate to ioredis - #427

Open
NuhYlmz wants to merge 3 commits into
cult-of-coders:masterfrom
NuhYlmz:monochat
Open

Add heap-based LIMIT_SORT driver and migrate to ioredis#427
NuhYlmz wants to merge 3 commits into
cult-of-coders:masterfrom
NuhYlmz:monochat

Conversation

@NuhYlmz

@NuhYlmz NuhYlmz commented Jul 9, 2026

Copy link
Copy Markdown

Summary

  • Replace the legacy redis npm package with ioredis 5.4.1
  • Update Redis connection config and add REDIS_OPLOG_URL environment variable support
  • Add heap-based LIMIT_SORT handling ported from Meteor's native OplogObserveDriver
  • Require Meteor 3.4 and the binary-heap package
  • Bump package version to 3.0.2

Problem

Publications with limit + sort use the LIMIT_SORT strategy. Previously, almost every insert, update, or remove triggered a full requery() — a sorted, limited MongoDB fetchAsync on every Redis event, on top of the findOneAsync already needed to load the changed document. Under high write volume, if that sorted fetch is slow (large collection, missing index, complex selector), events are processed sequentially and each one blocks until the requery completes. This causes a growing backlog and subscriptions falling behind real-time changes.

Solution

This implementation is based on Meteor's native oplog tailing driver (OplogObserveDriver), which uses the same heap-based approach for limit + sort queries. The goal is behavioral parity with Meteor's built-in oplog implementation, not a custom optimization.

For limit + sort queries (without skip), we maintain two in-memory heaps matching Meteor's design:

  • _published (MaxHeap) — the top-N documents currently sent to clients
  • _unpublishedBuffer (MinMaxHeap) — the next N candidates

The heap lifecycle (_addPublished, _removePublished, _addMatching, _removeMatching, _handleDocChange) and the fallback requery path (_fallbackRequeryLimitSort) mirror the corresponding methods in Meteor's OplogObserveDriver.

Instead of re-running a full sorted fetch on every change, a single findOneAsync for the changed document (already performed when the Redis event is received) is sufficient for most events. The heap uses that document together with the in-memory published set and buffer to decide whether to publish, buffer, promote, or evict — all via O(log n) comparator operations, with no additional MongoDB round-trip. A full fetch is only needed on initial subscription setup and in rare fallback cases when buffer state becomes uncertain (_fallbackRequeryLimitSort, fetching 2 × limit projected documents).

Breaking changes

  • Minimum Meteor version raised to 3.4
  • retryIntervalMs setting removed from config and README

jdgjsag67251 and others added 3 commits October 28, 2025 11:12
…roved compatibility. Added new lodash dependencies and removed obsolete references to _redisOplogCollectionMethods in ObservableCollection and RedisSubscriptionManager.
Use MaxHeap/MinMaxHeap for published and unpublished buffers to avoid full
requeries on eligible inserts/updates/removes, require Meteor 3.4 and
binary-heap, and fall back to requery when skip is present or heaps diverge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants