feat: enhance deduplicateItems with cross-batch/execution tracking #144

restareaByWeezy · 2026-01-22T15:37:35Z

🎯 Changes

This PR enhances the deduplicateItems option to support cross-batch/execution duplicate detection, simplifying the API by consolidating all deduplication functionality under a single option.

What Changed

Batcher & Queuer Enhancements:

✅ deduplicateItems now automatically tracks processedKeys across batches/executions
✅ Added maxTrackedKeys option (default: 1000) with FIFO eviction
✅ Enhanced onDuplicate callback with existingItem parameter to distinguish duplicate types:
- existingItem === undefined: cross-batch/execution duplicate (already processed)
- existingItem !== undefined: in-batch/queue duplicate
✅ Added clearProcessedKeys() method for manual key management

Usage Example

const batcher = new Batcher(processFn, {
  deduplicateItems: true,  // Single option enables both in-batch and cross-batch deduplication
  maxTrackedKeys: 100,
  getItemKey: (item) => item.id,
  onDuplicate: (newItem, existingItem, batcher) => {
    if (existingItem === undefined) {
      console.log('Already processed in previous batch')
    } else {
      console.log('Duplicate in current batch')
    }
  }
})

batcher.addItem({ id: 1, data: 'first' })
batcher.addItem({ id: 1, data: 'second' })  // in-batch duplicate
await batcher.flush()
batcher.addItem({ id: 1, data: 'third' })   // cross-batch duplicate (skipped)

// Clear processed keys when needed (e.g., for persistence restoration)
batcher.clearProcessedKeys()

Design Rationale

This follows the pattern of RateLimiter's executionTimes tracking:

Single option to enable the feature
Automatic state management
Persistence support through state serialization
Clear and intuitive API

Test Coverage

Batcher: 51 new test cases (109 total passing ✅)
Queuer: 58 new test cases (553 total passing ✅)
Comprehensive coverage of:
- Cross-batch/execution duplicate detection
- In-batch/queue deduplication with keep-first/keep-last strategies
- maxTrackedKeys FIFO eviction behavior
- clearProcessedKeys() method
- Persistence scenarios
- onDuplicate callback invocations

Documentation & Examples

Added examples/react/useBatcherDedup demonstrating:
- Cross-batch duplicate tracking visualization
- processedKeys state display
- maxTrackedKeys and clearProcessedKeys() usage
- Visual indicators for duplicate types
Updated changeset with detailed migration guide

✅ Checklist

I have followed the steps in the Contributing guide.
I have tested this code locally with pnpm run test:pr.

🚀 Release Impact

This change affects published code, and I have generated a changeset.
This change is docs/CI/dev-only (no release).

Changeset: .changeset/add-deduplication-feature.md

- Remove trackProcessedKeys option in favor of deduplicateItems - Add automatic processedKeys tracking when deduplicateItems is enabled - Add maxTrackedKeys option (default: 1000) with FIFO eviction - Enhance onDuplicate callback with existingItem parameter - existingItem is undefined for cross-batch duplicates - existingItem is the item for in-batch duplicates - Remove onSkip callback (consolidated into onDuplicate) - Remove skippedCount from state (no longer needed) - Support both cross-batch and in-batch deduplication - Add clearProcessedKeys() method for key management This change simplifies the API by making deduplicateItems the single option for both in-batch and cross-batch deduplication, following the pattern established by RateLimiter's executionTimes tracking.

- Add 51 new test cases covering deduplication scenarios - Test cross-batch duplicate detection and skipping - Test in-batch deduplication with keep-first/keep-last strategies - Test onDuplicate callback with correct parameters - Test maxTrackedKeys with FIFO eviction behavior - Test clearProcessedKeys() method - Test persistence scenarios with processedKeys - Verify processedKeys state management - Test interaction between cross-batch and in-batch deduplication All 109 batcher tests passing.

Apply the same deduplication pattern as Batcher: - Enhance deduplicateItems to enable automatic processedKeys tracking - Add maxTrackedKeys option with FIFO eviction (default: 1000) - Enhance onDuplicate callback with existingItem parameter - Support both cross-execution and in-queue deduplication - Add clearProcessedKeys() method - Add 58 comprehensive test cases The Queuer now follows the same simplified API pattern as Batcher, where deduplicateItems is the single option controlling both in-queue and cross-execution deduplication. All 553 queuer tests passing.

- Add useBatcherDedup React example demonstrating cross-batch deduplication - Show processedKeys tracking in UI - Demonstrate maxTrackedKeys and clearProcessedKeys() usage - Add visual indicators for cross-batch vs in-batch duplicates - Include comprehensive README with usage examples - Add changeset documenting the new deduplication features The example shows how deduplicateItems now automatically handles both in-batch and cross-batch deduplication with a single option, making it easier to prevent duplicate processing across multiple batch executions.

changeset-bot · 2026-01-22T15:37:40Z

🦋 Changeset detected

Latest commit: 99708fc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@tanstack/pacer	Minor
@tanstack/preact-pacer	Patch
@tanstack/react-pacer	Patch
@tanstack/solid-pacer	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

nx-cloud · 2026-01-28T02:31:21Z

View your CI Pipeline Execution ↗ for commit 99708fc

Command	Status	Duration	Result
`nx run-many --targets=build --exclude=examples/**`	✅ Succeeded	15s	View ↗

☁️ Nx Cloud last updated this comment at 2026-01-28 02:39:14 UTC

pkg-pr-new · 2026-01-28T02:34:55Z

More templates

@tanstack/pacer

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/pacer@144

@tanstack/pacer-devtools

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/pacer-devtools@144

@tanstack/pacer-lite

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/pacer-lite@144

@tanstack/preact-pacer

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/preact-pacer@144

@tanstack/preact-pacer-devtools

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/preact-pacer-devtools@144

@tanstack/react-pacer

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/react-pacer@144

@tanstack/react-pacer-devtools

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/react-pacer-devtools@144

@tanstack/solid-pacer

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/solid-pacer@144

@tanstack/solid-pacer-devtools

npm i https://pkg.pr.new/TanStack/pacer/@tanstack/solid-pacer-devtools@144

commit: 99708fc

KevinVandy · 2026-01-28T02:46:36Z

This is a well-formed PR, I'm just concerned that it adds too much bloat for a feature that can be implemented with an external Set<String> that you can set up in your own scope. I'm exploring if there can be a simpler implementation.

restareaByWeezy · 2026-01-28T09:00:15Z

@KevinVandy

That’s a fair point — thanks for calling it out!

I’m happy to revisit this with a simpler approach or move it out of the core if that better fits the library’s direction.
Out of curiosity, do you have any guiding principles around API surface area or feature inclusion that you use when reviewing PRs like this?
It would be really helpful context for future contributions.

restareaByWeezy added 4 commits January 23, 2026 00:32

restareaByWeezy and others added 3 commits January 27, 2026 23:33

chore: update lockfile

5013fe3

Merge branch 'main' into feat-deduplication

443d5f5

fix sherif and format

be7804d

ci: apply automated fixes

99708fc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: enhance deduplicateItems with cross-batch/execution tracking #144

feat: enhance deduplicateItems with cross-batch/execution tracking #144

Uh oh!

restareaByWeezy commented Jan 22, 2026 •

edited

Loading

Uh oh!

changeset-bot bot commented Jan 22, 2026 •

edited

Loading

Uh oh!

nx-cloud bot commented Jan 28, 2026 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Jan 28, 2026

Uh oh!

KevinVandy commented Jan 28, 2026

Uh oh!

restareaByWeezy commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

feat: enhance deduplicateItems with cross-batch/execution tracking #144

Are you sure you want to change the base?

feat: enhance deduplicateItems with cross-batch/execution tracking #144

Uh oh!

Conversation

restareaByWeezy commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 Changes

What Changed

Usage Example

Design Rationale

Test Coverage

Documentation & Examples

✅ Checklist

🚀 Release Impact

Uh oh!

changeset-bot bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

nx-cloud bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jan 28, 2026

Uh oh!

KevinVandy commented Jan 28, 2026

Uh oh!

restareaByWeezy commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

restareaByWeezy commented Jan 22, 2026 •

edited

Loading

changeset-bot bot commented Jan 22, 2026 •

edited

Loading

nx-cloud bot commented Jan 28, 2026 •

edited

Loading