WIP fix to duplicate inserts #1052
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When subscribing to a collection with includeInitialState: false, delete events were being filtered out because the sentKeys set was empty. This affected live queries with limit/offset where users would subscribe to get future changes after already loading initial data via preload() or values(). Changes: - Add skipFiltering flag separate from loadedInitialState to allow filtering to be skipped while still allowing requestSnapshot to work - Call markAllStateAsSeen() when includeInitialState is explicitly false - Change internal subscriptions to not pass includeInitialState: false explicitly, so they can be distinguished from user subscriptions - Add tests for optimistic delete behavior with limit Fixes the issue where deleted items would not disappear from live queries when using .limit() and subscribing with includeInitialState: false.
This is a DEBUG BUILD with [TanStack-DB-DEBUG] logs to help track down why delete events may not be reaching subscribers when using limit/offset. The debug logs cover: - subscribeChanges: when subscriptions are created - emitEvents: when events are emitted to subscriptions - Subscription.emitEvents: when individual subscriptions receive events - filterAndFlipChanges: when events are filtered or passed through - recomputeOptimisticState: when optimistic state is recomputed and events emitted - sendChangesToPipeline: when changes flow through the D2 pipeline - applyChanges: when D2 pipeline outputs to the live query collection To use: Filter browser console for "[TanStack-DB-DEBUG]" Also includes the fix for includeInitialState: false not emitting deletes.
Add comprehensive debug logging to: - createFilteredCallback in change-events.ts for whereExpression filtering - sendChangesToInput for D2 pipeline input - subscribeToOrderedChanges for orderBy/limit path - splitUpdates for update event handling - recomputeOptimisticState for pending sync key filtering This additional logging helps track where delete events may be filtered out when using live queries with limit/offset and where clauses.
Add debug logging to track: - scheduleGraphRun: when graph run is scheduled - executeGraphRun: when graph run executes or returns early - maybeRunGraph: when graph actually runs, pending work status This helps diagnose issues where deletes are sent to D2 pipeline but never appear in the output (applyChanges not called).
Add debug logging to track: - ReduceOperator: input processing, key handling, and result output - topK: consolidation, sorting, slicing, and result details Also add two new test cases: 1. Test delete from different page (page 1 delete while viewing page 2) - Verifies items shift correctly when delete occurs on earlier page 2. Test delete beyond TopK window (no-op case) - Verifies deleting item outside window doesn't affect results These tests and debug logs will help diagnose issues where deleted items don't disappear from live queries when using limit/offset.
Add additional debug logging to help diagnose delete issues: D2 graph (d2.ts): - Log when run() starts and completes with step count - Log pendingWork() results with operator IDs - Log when operators have pending work in step() Output operator (output.ts): - Log when run is called with message count - Log items in each message being processed Subscription (subscription.ts): - Log trackSentKeys with keys being added - Show total sentKeys count This should help diagnose scenarios where delete events are sent to D2 but no applyChanges output is produced.
Add operatorType property to Operator base class and log it when operators run. This will help identify which operators are processing the delete and where the data is being lost. Also add detailed logging to LinearUnaryOperator.run() to show: - Input message count - Input/output item counts - Sample of input and output items This should reveal exactly which operator is dropping the delete.
This is the key operator for orderBy+limit queries. Add detailed logging to: - run(): Show message count and index size - processElement(): Show key, multiplicity changes, and action (INSERT/DELETE/NO_CHANGE) - processElement result: Show moveIn/moveOut keys This should reveal exactly why deletes aren't producing output changes when the item exists in the TopK index.
…ity issues When an item is inserted multiple times without a delete in between, D2 multiplicity goes above 1. Then when a single delete arrives, multiplicity goes from 2 to 1 (not 0), so TopK doesn't emit a DELETE event. This fix: 1. Filters out duplicate inserts in filterAndFlipChanges when key already in sentKeys 2. Removes keys from sentKeys on delete in both filterAndFlipChanges and trackSentKeys 3. Updates test expectation to reflect correct behavior (2 events instead of 3) Root cause: Multiple subscriptions or sync mechanisms could send duplicate insert events for the same key, causing D2 to track multiplicity > 1.
The previous fix in CollectionSubscription.filterAndFlipChanges was only catching duplicates at the subscription level. But each live query has its own CollectionSubscriber with its own D2 pipeline. This fix adds a sentToD2Keys set in CollectionSubscriber to track which keys have been sent to the D2 input, preventing duplicate inserts at the D2 level regardless of which code path triggers them. Also clears the tracking on truncate events.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Forked from earlier state of #1044