Critical Performance Issue: payload.find Takes Significantly Longer Than Mongoose find for Large Data Sets

[nihalnclt]

Link to reproduction

No response

Describe the Bug

I am experiencing a significant performance issue when using Payload CMS's payload.find to fetch news documents that include populated fields (category, subcategory).

• Fetching 1,000 documents using payload.find takes ~4 seconds
• Fetching the same 1,000 documents using Mongoose's find (with .populate()) takes ~600ms

This issue becomes a major bottleneck when handling large datasets.

Expected Behavior:

Fetching populated data using payload.find should perform similarly to Mongoose's find().populate(), rather than being 15x slower.

Actual Behavior:

• payload.find takes ~4 seconds to fetch 1,000 news documents with populated category and subcategory.
• Mongoose’s find().populate() fetches the same data in ~600ms.

Environment:

• Payload CMS version: (2.30.0)
• Node.js version: (23.6.1)
• Database Mongodb

To Reproduce

1. Setup:

• A news collection with at least 1,000 documents
• Each document has two populated fields
  - category (reference to a category collection)
  - subcategory (reference to a subcategory collection)

2. Fetch 1,000 documents using payload.find with population. (Time taken: ~4s)

const result = await payload.find({ collection: 'news', limit: 1000, depth: 2, });

3. Fetch 1,000 documents using Mongoose’s find with .populate(). (Time taken: ~600ms)

const result = await payload.db.collections["news"] .limit(1000) .populate('category') .populate('subcategory');

Payload Version

2.30.0

Adapters and Plugins

db-mongodb

Screenshots

[DanRibbens]

We have made a lot of performance enhancements in v3 and I would expect that this has improved quite a bit if you tried similar benchmarking against latest. That said, there are major differences between fetching from mongodb directly and I'm not surprised that there is a time difference between these two.

The Payload find operation will get the document which will have the id of the relationship fields and then in our field hooks we call a dataloader pattern. That means that it will go back to the database along with any access control needed for populating docs and run all collection and field hooks of the operation and localizing data for every document. In contrast, when you query the DB directly the documents populate in one request and no hooks or any other code is called.

With the way the dataloader pattern works it matters if you're populating the same doc in multiple places or getting different docs 1000 times.

I'd be curious what kind of perf you see in v3 and especially if you make use of defaultPopulate and other new features for perf improvements.

I'm converting this to a discussion and if there are fixes that come out of this conversation we can create new issues for those action items.