indexeddb: improve allDocs() perf with skip & key ranges #8603

alxndrsn · 2023-02-15T13:39:13Z

This improves pouchdb-adapter-indexeddb performance for key ranges by:

adding a new index ([deleted, id]) to allow for skipping deleted docs by default
moving _local/ docs to the META_STORE to simplify allDocs() implementation

TODO

batching in allDocs()
add _getLocal() implementation
rename META_STORE to META_LOCAL_STORE
check if getRevisionTree()/doCompaction() need to support local docs
resolve trade-off comments
resolve TODOs
update benchmarks once everything else is done
update perf test with: idb-master vs indexeddb-master vs idb-this-pr

Benchmarks

The standard performance test suite was run with:

Client

  "client": {
    "browser": {
      "name": "Chrome Headless",
      "version": "108.0.5359.29",
      "major": "108"
    },
    "device": {},
    "engine": {
      "name": "Blink",
      "version": "108.0.5359.29"
    },
    "cpu": {
      "architecture": "amd64"
    },
    "os": {
      "name": "Linux",
      "version": "x86_64"
    },
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36"
  }

Adapters

idb - pouchdb-adapter-idb as on the current master (30fd84f)
master - pouchdb-adapter-indexeddb as on the current master
this-pr - pouchdb-adapter-indexeddb as on this branch

Results

The full suite was run 51 times, and the lowest median score for each test was taken for each adapter.

Key:

Scores are in milliseconds; lower is better.
The winning score is marked with !.
Where another score is within 5% of the winning score, it is marked with ~.

             |                                 |          idb |       master |      this-pr
-------------|---------------------------------|--------------|--------------|--------------
 basics      | basic-inserts                   |       2.10   |       1.70 ! |       1.80  
             | bulk-inserts                    |      49.35   |      29.20 ! |      32.80  
             | bulk-inserts-large-docs         |     116.00 ~ |     111.80 ! |     112.15 ~
             | bulk-inserts-massive-docs       |    2067.70 ~ |    2054.75 ! |    2059.40 ~
             | basic-updates                   |      78.90 ! |     137.65   |     149.65  
             | basic-gets                      |       1.40   |       1.10 ! |       1.10 !
             | all-docs-skip-limit             |      88.25   |     208.00   |      65.10 !
             | all-docs-startkey-endkey        |      14.55 ! |      28.70   |      17.20  
             | all-docs-keys                   |       8.20   |       4.25 ~ |       4.10 !
             | all-docs-include-docs           |      16.65   |      11.60   |       6.80 !
             | pull-replication-one-generation |    1614.60 ! |    1692.80 ~ |    1624.20 ~
             | pull-replication-two-generation |    1608.30 ! |    1697.40   |    1679.40 ~
-------------|---------------------------------|--------------|--------------|--------------
 views       | temp-views                      |    4603.00 ! |    7012.00   |    4792.30 ~
             | build-secondary-index           |     831.40   |    2020.60   |     785.60 !
             | persisted-views                 |      35.80   |      32.50   |      28.10 !
             | persisted-views-stale-ok        |      15.00   |      16.45   |      12.45 !
-------------|---------------------------------|--------------|--------------|--------------
 find        | create-index                    |   27509.80   |    2246.10   |    1244.50 !
             | simple-find-query               |      11.80   |       4.90 ! |       4.90 !
             | simple-find-query-no-index      |     842.10   |     462.00   |     282.00 !
             | complex-find-query              |    2131.20   |     529.80 ~ |     527.30 !
             | complex-find-query-no-index     |     829.30   |     443.80   |     296.90 !
             | multi-field-query               |    2124.60   |     519.70 ! |     528.10 ~
-------------|---------------------------------|--------------|--------------|--------------
 attachments | basic-attachments               |       8.90 ! |      10.00   |      10.20

garethbowen · 2023-02-19T17:40:52Z

@alxndrsn Awesome work! Thanks for the detailed reporting.

Lots of big improvements, and plenty that are break even.

The only one that really stands out is basic-updates. Any idea what's going on there?

alxndrsn · 2023-02-20T13:13:59Z

basic-updates. Any idea what's going on there?

Great question. The test is defined at:

https://github.com/pouchdb/pouchdb/blob/30fd84f1ae88ca8687c2842bd131e67e7cf7c41c/tests/performance/perf.basics.js#L101-L121

So it's measuring bulkDocs() and allDocs(), which is surprising. I'll look into this further.

alxndrsn · 2023-10-11T17:43:14Z

Added new test for local docs with attachments at #8785; maybe useful for this PR as it will separate storage of local docs from other docs.

garethbowen

Love it.

There are quite a lot of TODO and REVIEW comments which should be addressed one way or the other.

Some ideas inline...

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js

packages/node_modules/pouchdb-adapter-indexeddb/src/bulkDocs.js

packages/node_modules/pouchdb-adapter-indexeddb/src/get.js

packages/node_modules/pouchdb-adapter-indexeddb/src/index.js

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js

This reverts commit 8b95c7f.

Fixes error: An error occurred while bailing: TypeError: Cannot read property 'forEach' of undefined

Fixes error: An error occurred while bailing: TypeError: Cannot read property 'forEach' of undefined And subsequent error: An error occurred while bailing: TypeError: test.titlePath is not a function

…alldocs

packages/node_modules/pouchdb-adapter-indexeddb/src/bulkDocs.js

…alldocs

This reverts commit f86dd92.

This reverts commit f889598.

SourceR85

The logical changes look good to me.

Would be nice, if @garethbowen could take a final look at these changes.

tests/integration/test.all_docs.js

packages/node_modules/pouchdb-adapter-indexeddb/src/util.js

packages/node_modules/pouchdb-adapter-indexeddb/src/setup.js

garethbowen

Just readability fixes - the rest looks great.

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js

garethbowen · 2024-04-01T11:11:06Z

@alxndrsn Any idea why this change makes the views perform worse?

views       | temp-views                      |         5099.90 ! |         6630.30       
-           | build-secondary-index           |         1547.10 ! |         1847.30       
-           | persisted-views                 |           33.90 ! |           36.45       
-           | persisted-views-stale-ok        |           17.25 ~ |           16.95 ~

…alldocs

alxndrsn · 2024-04-03T07:07:05Z

@alxndrsn Any idea why this change makes the views perform worse?

Great question, looking into this.

alxndrsn · 2024-04-03T11:15:27Z

@garethbowen I've re-run perf test suites basic and temp-views with Chromium 108, which looks to have been the version used for the results in the original PR message.

Branches used represent master and this PR at the time of opening this PR, and master and this PR in the present day:

Using stat: median
Client: Chrome Headless 108
Adapter: indexeddb
Versions tested:
1. 30fd84f - master - historic
2. acc12f6 - master - today
3. 18f706f - this PR - today
4. c361764 - this PR - historic

-           | -                               | itr |         #1 | itr |         #2 | itr |         #3 | itr |         #4
-------------------------------------------------------------------------------------------------------------------------
basics      | basic-inserts                   |  22 |     1.80 ~ |  22 |     1.80 ~ |  22 |     1.80 ~ |  22 |     1.80 ~
-           | bulk-inserts                    |  22 |    34.22 ~ |  22 |    34.80 ~ |  22 |    61.75   |  22 |    60.92  
-           | bulk-inserts-large-docs         |  22 |   119.32 ~ |  22 |   124.70 ~ |  22 |   180.17   |  22 |   180.62  
-           | bulk-inserts-massive-docs       |  22 |  2091.82 ~ |  22 |  2083.63 ~ |  22 |  2076.68 ~ |  22 |  2094.50 ~
-           | basic-updates                   |  22 |   154.25 ~ |  22 |   148.60 ~ |  22 |   159.23   |  22 |   159.07  
-           | basic-gets                      |  22 |     1.10 ~ |  22 |     1.10 ~ |  22 |     1.10 ~ |  22 |     1.10 ~
-           | all-docs-skip-limit             |  22 |   218.67   |  22 |   217.00   |  22 |    73.25   |  22 |    67.57 !
-           | all-docs-startkey-endkey        |  22 |    31.25   |  22 |    31.38   |  22 |    18.13 ~ |  22 |    18.38 ~
-           | all-docs-keys                   |  22 |     4.65   |  22 |     4.50 ~ |  22 |     4.40 ~ |  22 |     4.40 ~
-           | all-docs-include-docs           |  22 |    12.35   |  22 |    12.40   |  22 |     8.80   |  22 |     7.13 !
-           | pull-replication-one-generation |  22 |  1778.05 ~ |  22 |  1730.90 ~ |  22 |  1822.25   |  22 |  1817.10 ~
-           | pull-replication-two-generation |  22 |  1830.80 ~ |  22 |  1825.25 ~ |  22 |  1844.70 ~ |  22 |  1852.30 ~
-------------------------------------------------------------------------------------------------------------------------
views       | temp-views                      |  22 |  7033.05 ! |  22 |  7488.70   |  22 | 11233.55   |  22 | 11202.45  
-           | build-secondary-index           |  22 |  2345.30 ~ |  22 |  2301.40 ~ |  22 |  3051.20   |  22 |  3415.40  
-           | persisted-views                 |  22 |    36.00   |  22 |    38.20   |  22 |    39.98   |  22 |    34.05 !
-           | persisted-views-stale-ok        |  22 |    18.30   |  22 |    18.50   |  22 |    20.03   |  22 |    16.17 !

  * itr = test iterations

It shows slowdown for this branch for temp-views, likely because the temp views involve a massive amount of bulkDocs() calls, which shows slowdown in the basics suite.

What's weird is that the original results shared in the PR showed so much improvement in temp-views. I don't have a satisfactory explanation for that.

The slowdown is cause for some concern, but perhaps it doesn't matter? Pouch docs read:

Temporary queries are very slow, and we only recommend them for quick debugging during development.

https://pouchdb.com/guides/queries.html#mappin-and-reducin

garethbowen · 2024-04-03T11:32:12Z

I agree that temp views aren't a big concern (at least for me!) but build-secondary-index is, right? The initial results had this beating master and idb and now it's gone the other way.

alxndrsn · 2024-04-03T16:54:18Z

I agree that temp views aren't a big concern (at least for me!) but build-secondary-index is, right? The initial results had this beating master and idb and now it's gone the other way.

As above, I'm also confused by this. I think I have a separate fix which reduces the impact significantly... but also applies to the master version 😁 Benchmarking in progress...

…alldocs

alxndrsn · 2024-04-04T06:23:54Z

@garethbowen I've deleted the seq index from map-reduce databases, as it's not required there. This seems to have reduced the impact. However, it's worth noting this change could also be applied to the current master, with significant effect:

Using stat: min
Client: Chrome Headless 123
Adapters:
1. indexeddb - 5bc8f31 - master
2. indexeddb - 6e52efe - master with seq index
3. indexeddb - a8b0c63 - this pr

-           | -                               | itr |         #1 | itr |         #2 | itr |         #3
------------------------------------------------------------------------------------------------------
basics      | basic-inserts                   |  20 |     1.20 ~ |  20 |     1.20 ~ |  20 |     1.40  
-           | bulk-inserts                    |  20 |    23.15 ~ |  20 |    23.55 ~ |  20 |    27.30  
-           | bulk-inserts-large-docs         |  20 |    36.45   |  20 |    33.00 ! |  20 |    77.40  
-           | bulk-inserts-massive-docs       |  20 |  1766.65 ~ |  20 |  1767.65 ~ |  20 |  1763.15 ~
-           | basic-updates                   |  20 |   114.80 ~ |  20 |   118.55 ~ |  20 |   120.80 ~
-           | basic-gets                      |  20 |     1.10 ~ |  20 |     1.10 ~ |  20 |     1.10 ~
-           | all-docs-skip-limit             |  20 |   190.15   |  20 |   191.65   |  20 |    54.45 !
-           | all-docs-startkey-endkey        |  20 |    29.30   |  20 |    29.30   |  20 |    15.85 !
-           | all-docs-keys                   |  20 |     3.60 ~ |  20 |     3.70   |  20 |     3.50 ~
-           | all-docs-include-docs           |  20 |    11.00   |  20 |    10.95   |  20 |     7.30 !
-           | pull-replication-one-generation |  20 |  1710.70 ~ |  20 |  1738.60 ~ |  20 |  1783.50 ~
-           | pull-replication-two-generation |  20 |  1720.20 ~ |  20 |  1734.40 ~ |  20 |  1760.10 ~
------------------------------------------------------------------------------------------------------
views       | temp-views                      |  20 |  5130.60   |  20 |  4182.90 ! |  20 |  6053.70  
-           | build-secondary-index           |  20 |  1556.80   |  20 |  1151.30 ! |  20 |  1705.60  
-           | persisted-views                 |  20 |    33.70 ~ |  20 |    33.20 ~ |  20 |    33.75 ~
-           | persisted-views-stale-ok        |  20 |    16.45 ~ |  20 |    15.85 ~ |  20 |    16.70  
------------------------------------------------------------------------------------------------------
find        | create-index                    |  20 |  1530.80   |  20 |  2349.80   |  20 |  1148.20 !
-           | simple-find-query               |  20 |     5.10   |  20 |     4.40 ! |  20 |     4.80  
-           | simple-find-query-no-index      |  20 |   408.30   |  20 |   424.00   |  20 |   307.10 !
-           | complex-find-query              |  20 |   461.10 ~ |  20 |   462.40 ~ |  20 |   463.10 ~
-           | complex-find-query-no-index     |  20 |   399.60   |  20 |   396.20   |  20 |   311.10 !
-           | multi-field-query               |  20 |   461.70 ~ |  20 |   453.70 ~ |  20 |   465.30 ~
------------------------------------------------------------------------------------------------------
attachments | basic-attachments               |  20 |     8.30 ~ |  20 |     8.35 ~ |  20 |     8.60 ~
-           | many-attachments-base64         |  20 |    55.65 ~ |  20 |    56.45 ~ |  20 |    55.75 ~
-           | many-attachments-binary         |  20 |     5.10 ~ |  20 |     5.10 ~ |  20 |     5.00 ~

  * itr = test iterations

…alldocs

alxndrsn · 2024-04-04T12:20:35Z

Using stat: min
Client: Chrome Headless 123
Adapters:
1. idb - 9ba277b - master
2. indexeddb - 9ba277b - master
3. indexeddb - 4fd2f51 - this pr

-           | -                               | itr |         #1 | itr |         #2 | itr |         #3
------------------------------------------------------------------------------------------------------
basics      | basic-inserts                   |  10 |     1.60   |  10 |     1.20 ! |  10 |     1.30  
-           | bulk-inserts                    |  10 |    53.45   |  10 |    23.55 ! |  10 |    27.45  
-           | bulk-inserts-large-docs         |  10 |   139.60   |  10 |    41.20 ! |  10 |    64.40  
-           | bulk-inserts-massive-docs       |  10 |  1769.70 ~ |  10 |  1762.70 ~ |  10 |  1772.45 ~
-           | basic-updates                   |  10 |   116.30 ~ |  10 |   118.30 ~ |  10 |   126.30  
-           | basic-gets                      |  10 |     1.40   |  10 |     1.10 ~ |  10 |     1.10 ~
-           | all-docs-skip-limit             |  10 |    78.60   |  10 |   190.65   |  10 |    55.00 !
-           | all-docs-startkey-endkey        |  10 |    13.30 ! |  10 |    28.90   |  10 |    15.55  
-           | all-docs-keys                   |  10 |     6.80   |  10 |     3.60 ~ |  10 |     3.60 ~
-           | all-docs-include-docs           |  10 |    15.00   |  10 |    11.35   |  10 |     7.25 !
-           | pull-replication-one-generation |  10 |  1545.60 ! |  10 |  1762.70   |  10 |  1787.60  
-           | pull-replication-two-generation |  10 |  1581.40 ! |  10 |  1739.60   |  10 |  1790.70  
------------------------------------------------------------------------------------------------------
views       | temp-views                      |  10 |  3954.00 ! |  10 |  5126.90   |  10 |  6281.80  
-           | build-secondary-index           |  10 |   664.60 ! |  10 |  1527.50   |  10 |  1614.20  
-           | persisted-views                 |  10 |    32.45 ~ |  10 |    33.55 ~ |  10 |    33.60 ~
-           | persisted-views-stale-ok        |  10 |    13.85 ! |  10 |    16.30   |  10 |    16.45  
------------------------------------------------------------------------------------------------------
find        | create-index                    |  10 | 19660.30   |  10 |  1714.30   |  10 |  1069.10 !
-           | simple-find-query               |  10 |    10.30   |  10 |     5.00 ! |  10 |     5.60  
-           | simple-find-query-no-index      |  10 |   736.20   |  10 |   415.60   |  10 |   312.70 !
-           | complex-find-query              |  10 |  1813.50   |  10 |   458.30 ~ |  10 |   471.80 ~
-           | complex-find-query-no-index     |  10 |   719.40   |  10 |   399.60   |  10 |   314.50 !
-           | multi-field-query               |  10 |  1909.80   |  10 |   455.70 ~ |  10 |   463.00 ~
------------------------------------------------------------------------------------------------------
attachments | basic-attachments               |  10 |     7.10 ! |  10 |     8.30   |  10 |     8.60  
-           | many-attachments-base64         |  10 |    61.30   |  10 |    55.15 ~ |  10 |    54.45 ~
-           | many-attachments-binary         |  10 |     7.50   |  10 |     5.10 ~ |  10 |     5.10 ~

  * itr = test iterations

garethbowen

One naming suggestion, and two left over suggestions from a previous review, then I think this is ready to go.

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js

Style change is postponed

…alldocs

garethbowen

Ready when you are. Great work on this!

alxndrsn · 2024-04-09T11:13:00Z

@garethbowen I've made a small update to getAttachment()'s handling of local docs to bring it more into line with the idb and memory adapters. I've also added a couple of extra tests around this functionality.

Please note that this marks a change from the behaviour of the indexeddb adapter in master - getAttachment() would previously return attachments on local docs iff using the indexeddb adapter. This behaviour did not seem to be intentional - as far as I can tell, correct behaviour/support of attachments on local docs is not specified in PouchDB.

alxndrsn added 2 commits February 15, 2023 13:37

indexeddb: improve allDocs() perf with skip & key ranges

c361764

Rename index

e0dfb7a

alxndrsn and others added 6 commits August 30, 2023 14:30

Merge branch 'master' into indexeddb-faster-alldocs

6c9f86d

Merge branch 'master' into indexeddb-faster-alldocs

dbe2405

Merge branch 'master' into indexeddb-faster-alldocs

44a0a15

Merge branch 'master' into indexeddb-faster-alldocs

6eb3972

fix eol-last

9bc7927

add migration

42cc853

alxndrsn marked this pull request as ready for review October 11, 2023 17:46

garethbowen suggested changes Oct 16, 2023

View reviewed changes

alxndrsn and others added 15 commits October 16, 2023 09:57

Use isLocalId()

dd6ed4c

Merge branch 'master' into indexeddb-faster-alldocs

796e1e4

Merge branch 'master' into indexeddb-faster-alldocs

f470aa7

rename META_STORE to META_LOCAL_STORE

8d72af6

get(): use isLocalId()

7f3e980

Implement getLocal()

8b95c7f

Revert "Implement getLocal()"

7d468da

This reverts commit 8b95c7f.

Implement getLocal()

e2ea6ee

Merge branch 'master' into indexeddb-faster-alldocs

1cde95c

mocha v10: fix failure handling

2187822

Fixes error: An error occurred while bailing: TypeError: Cannot read property 'forEach' of undefined

mocha v10: fix failure handling

d50645d

Fixes error: An error occurred while bailing: TypeError: Cannot read property 'forEach' of undefined And subsequent error: An error occurred while bailing: TypeError: test.titlePath is not a function

indexeddb._getLocal(): process attachments

85fbca8

Merge branch 'master' into indexeddb-faster-alldocs

dec066a

more comment

02a837c

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

ad652f8

…alldocs

alxndrsn commented Mar 24, 2024

View reviewed changes

packages/node_modules/pouchdb-adapter-indexeddb/src/bulkDocs.js Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

1ec03cd

…alldocs

alxndrsn added 4 commits March 27, 2024 16:44

Revert "getLocal(): remove use strict & attachment support"

dd62ea2

This reverts commit f86dd92.

Revert "lint"

213d8c4

This reverts commit f889598.

remove use strict

aed6f98

use const

56a17f4

SourceR85 previously requested changes Mar 28, 2024

View reviewed changes

garethbowen suggested changes Apr 1, 2024

View reviewed changes

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js Outdated Show resolved Hide resolved

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

18f706f

…alldocs

alxndrsn added 3 commits April 3, 2024 12:37

delete seq index; don't create other index

f9a5ba3

don't delete seq - seems to be required somewhere

cb90ef3

delete seq; maintain _isDeleted_id

179cc49

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

a8b0c63

…alldocs

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

4fd2f51

…alldocs

garethbowen suggested changes Apr 5, 2024

View reviewed changes

packages/node_modules/pouchdb-adapter-indexeddb/src/allDocs.js Outdated Show resolved Hide resolved

alxndrsn added 2 commits April 5, 2024 07:57

allDocs: don't recurse

8c0b977

rename deleted,id db index

063c725

alxndrsn added 2 commits April 8, 2024 11:45

Merge remote-tracking branch 'upstream/master' into indexeddb-faster-…

0b06fbc

…alldocs

new test: make more robust in CI

569557d

garethbowen approved these changes Apr 8, 2024

View reviewed changes

alxndrsn added 2 commits April 9, 2024 10:23

Update/add tests for local doc attachments

1412422

update github issue link

d8d9f42

alxndrsn merged commit 2789b70 into apache:master Apr 9, 2024

indexeddb: improve allDocs() perf with skip & key ranges #8603

indexeddb: improve allDocs() perf with skip & key ranges #8603

Uh oh!

Conversation

alxndrsn commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Benchmarks

Client

Adapters

Results

Uh oh!

garethbowen commented Feb 19, 2023

Uh oh!

alxndrsn commented Feb 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alxndrsn commented Oct 11, 2023

Uh oh!

garethbowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SourceR85 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

garethbowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

garethbowen commented Apr 1, 2024

Uh oh!

alxndrsn commented Apr 3, 2024

Uh oh!

alxndrsn commented Apr 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

garethbowen commented Apr 3, 2024

Uh oh!

alxndrsn commented Apr 3, 2024

Uh oh!

alxndrsn commented Apr 4, 2024

Uh oh!

alxndrsn commented Apr 4, 2024

Uh oh!

garethbowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

garethbowen left a comment

Choose a reason for hiding this comment

Uh oh!

alxndrsn commented Apr 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alxndrsn commented Feb 15, 2023 •

edited

Loading

alxndrsn commented Feb 20, 2023 •

edited

Loading

alxndrsn commented Apr 3, 2024 •

edited

Loading