Skip to content

Commit 6ae34f0

Browse files
tmmampagent
andauthored
docs(zero-rust): use the quiet orb's benchmark numbers (#175)
The earlier numbers came from an orb that ran both native clients 2.2–2.4 times slower, with JS on Node 24 instead of the pinned Node 22 (T-7e684b8b-74ff-4a5d-8321-d7f85b32c311). On the quiet orb Rust materializes 100,000 rows at the same cost per row as 10,000. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: Amp <amp@ampcode.com>
1 parent 7b663b3 commit 6ae34f0

1 file changed

Lines changed: 49 additions & 48 deletions

File tree

‎rust/BENCHMARKS.md‎

Lines changed: 49 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -37,56 +37,58 @@ gate on. CI builds them with the workspace (`cargo clippy --all-targets`,
3737

3838
## Results
3939

40-
Medians of three runs; lower is better. Rust and Swift ran in turns on one
41-
machine, an 8-core Intel Xeon at 2.6 GHz with 16 GB, Linux 6.1, with Rust
42-
1.99.1 (release build) and Swift 6.2.4 (`swift build -c release` of this
43-
checkout); JS ran just before, with Node 24.22 under vitest. The JS
44-
`zero.bench` and hash rows are from an earlier run on the same machine.
45-
46-
`swift/BENCHMARKS.md`'s Linux column has Swift 2–3 times faster than it ran
47-
here (materializing 100,000 rows in 7.07 ms, not 16.8 ms), from what it
48-
calls the same kind of machine; JS ran here as fast as that document has
49-
it; why isn't known yet. Against that column, Rust is behind Swift on
50-
materializing 100,000 rows (10.5 ms against 7.07 ms) and level with it on
51-
reading them as structs (23.8 ms against 24.8 ms).
52-
53-
In short: on this machine Rust is 2–5× faster than JS and 1.5–4.6× faster
54-
than Swift on views of 1,000 to 100,000 rows, 17× faster than JS on an
55-
optimistic update, and 20–40× on hashing (where vitest flatters Rust). It
56-
uses about a third of JS's memory and two thirds of Swift's.
40+
Lower is better. The rows without a mark are medians of five runs of
41+
`--time 2 memory hydration` taken in turns, rotating the order of the
42+
three clients, on a quiet Amp orb (an 8-core Intel Xeon at 2.6 GHz with
43+
15 GB), with Rust 1.99.1 (release build), Swift 6.2.4 (`swift build -c
44+
release`), and JS on Node 22.23, the version the repository pins, under
45+
vitest.
46+
47+
The rows marked † are medians of three runs from an earlier, slower orb,
48+
where Rust and Swift ran 2.2–2.4 times slower than on the quiet one and
49+
JS ran on Node 24.22, which runs these benchmarks about twice as fast as
50+
Node 22. Compare the clients within a † row, not with the other rows.
51+
Shared machines vary this much, so compare clients on the same machine, in
52+
turns.
53+
54+
In short: Rust is 6–9× faster than JS and 1.5–1.8× faster than Swift on
55+
views of 1,000 to 100,000 rows, and 1.7× faster than Swift reading them as
56+
structs. It uses about half of JS's memory and two thirds of Swift's. Each
57+
native client's cost per row is about the same at 10,000 and 100,000
58+
rows; JS's is 1.6 times as much at 100,000.
5759

5860
| Benchmark | JS (Node) | Rust | Swift (Linux) |
5961
| ------------------------------------------ | --------------------------- | ------------------------ | ------------------------ |
60-
| zero.bench: all 1000 rows x 10 columns | 236 µs | 52.1 µs | 91.6 µs |
61-
| zero.bench: pk = N | 11.0 µs | 4.60 µs | 21.3 µs |
62-
| zero.bench: lower 500 rows | 152 µs | 54.6 µs | 130 µs |
63-
| materialize 1,000 rows | 229 µs | 46.3 µs | 85.6 µs |
64-
| materialize 10,000 rows | 2.06 ms | 565 µs | 863 µs |
65-
| materialize 100,000 rows | 30.1 ms | 10.5 ms | 16.8 ms |
66-
| memory for 100,000 rows and a view of them | 239 MB | 83 MB | 127 MB |
67-
| optimistic update with a 1000-row view | 274 µs | 15.9 µs | 39.5 µs |
68-
| h32, 64 / 1024 / 8192 chars | 433 ns / 3.87 µs / 32.3 µs | 10 ns / 151 ns / 1.21 µs | 27 ns / 172 ns / 1.23 µs |
69-
| h64, 64 / 1024 / 8192 chars | 703 ns / 6.37 µs / 51.5 µs | 22 ns / 303 ns / 2.42 µs | 42 ns / 323 ns / 2.45 µs |
70-
| h128, 64 / 1024 / 8192 chars | 1.29 µs / 12.5 µs / 95.3 µs | 41 ns / 605 ns / 4.85 µs | 67 ns / 628 ns / 4.88 µs |
62+
| zero.bench: all 1000 rows x 10 columns † | 236 µs | 52.1 µs | 91.6 µs |
63+
| zero.bench: pk = N † | 11.0 µs | 4.60 µs | 21.3 µs |
64+
| zero.bench: lower 500 rows † | 152 µs | 54.6 µs | 130 µs |
65+
| materialize 1,000 rows | 279 µs | 42 µs | 74 µs |
66+
| materialize 10,000 rows | 2.74 ms | 470 µs | 715 µs |
67+
| materialize 100,000 rows | 43.1 ms | 4.64 ms | 7.04 ms |
68+
| memory for 100,000 rows and a view of them | 158 MB | 82.7 MB | 126 MB |
69+
| optimistic update with a 1000-row view † | 274 µs | 15.9 µs | 39.5 µs |
70+
| h32, 64 / 1024 / 8192 chars † | 433 ns / 3.87 µs / 32.3 µs | 10 ns / 151 ns / 1.21 µs | 27 ns / 172 ns / 1.23 µs |
71+
| h64, 64 / 1024 / 8192 chars † | 703 ns / 6.37 µs / 51.5 µs | 22 ns / 303 ns / 2.42 µs | 42 ns / 323 ns / 2.45 µs |
72+
| h128, 64 / 1024 / 8192 chars † | 1.29 µs / 12.5 µs / 95.3 µs | 41 ns / 605 ns / 4.85 µs | 67 ns / 628 ns / 4.88 µs |
7173

7274
As `hash.bench.ts` notes, vitest keeps imported functions from being
7375
inlined, so its JS hash numbers are pessimistic; bundled, they are a few
7476
times faster. The memory figures are the change in resident memory
75-
(`VmRSS`), from one run.
77+
(`VmRSS`).
7678

7779
Rust and Swift only:
7880

79-
| Benchmark | Rust | Swift (Linux) |
80-
| --------------------------------------------------------- | ------- | ------------- |
81-
| materialize 1,000 rows as structs | 121 µs | 240 µs |
82-
| materialize 10,000 rows as structs | 1.42 ms | 2.41 ms |
83-
| materialize 100,000 rows as structs | 23.8 ms | 37.4 ms |
84-
| poke putting 1 row, 1000-row view | 12.4 µs | 49.1 µs |
85-
| poke putting 100 rows, 1000-row view | 485 µs | 1.18 ms |
86-
| poke putting 1000 rows, 1000-row view | 5.01 ms | 10.8 ms |
87-
| round trip (update, push, confirming poke), 1000-row view | 39.7 µs | 118 µs |
88-
| first sync: zbugs poke of 2,058 rows, issue list view | 15.3 ms | 42.9 ms |
89-
| first sync: the same poke, no view | 8.86 ms | 24.4 ms |
81+
| Benchmark | Rust | Swift (Linux) |
82+
| ----------------------------------------------------------- | ------- | ------------- |
83+
| materialize 1,000 rows as structs † | 121 µs | 240 µs |
84+
| materialize 10,000 rows as structs | 1.15 ms | 1.98 ms |
85+
| materialize 100,000 rows as structs | 11.2 ms | 19.6 ms |
86+
| poke putting 1 row, 1000-row view † | 12.4 µs | 49.1 µs |
87+
| poke putting 100 rows, 1000-row view † | 485 µs | 1.18 ms |
88+
| poke putting 1000 rows, 1000-row view † | 5.01 ms | 10.8 ms |
89+
| round trip (update, push, confirming poke), 1000-row view † | 39.7 µs | 118 µs |
90+
| first sync: zbugs poke of 2,058 rows, issue list view † | 15.3 ms | 42.9 ms |
91+
| first sync: the same poke, no view † | 8.86 ms | 24.4 ms |
9092

9193
`first sync` replays zbugs' first sync as the zbugs-swift app got it, with
9294
the rows Swift's benchmark generates from the same seed: one poke of 21
@@ -129,21 +131,20 @@ timed from the poke's frames arriving to the view being flushed.
129131
column index and JS engines share hidden classes. The parser follows a
130132
recent object's keys while the next object has the same ones, so a
131133
synced row allocates neither its keys nor their list, and comparators
132-
and filters remember where their column was. Run in turns on one
133-
machine: the memory of 100,000 rows went from 181 to 83 MB,
134+
and filters remember where their column was. Run in turns on the
135+
slower orb: the memory of 100,000 rows went from 181 to 83 MB,
134136
materializing them from 20.9 to 10.5 ms and reading them as structs
135137
from 48.3 to 23.8 ms, 10,000 rows from 678 to 565 µs (as structs 1.99
136138
to 1.42 ms), a poke of 1000 rows from 7.74 to 5.01 ms, and a first
137139
sync from 17.0 to 15.3 ms. Materializing 100,000 rows went from 3.1
138-
times the cost per row of 10,000 to 1.9 times.
140+
times the cost per row of 10,000 to 1.9 times there; on the quiet orb
141+
it costs the same per row (46 ns at 100,000 rows, 47 ns at 10,000).
142+
The slower orb's extra cost at 100,000 rows is cache misses: callgrind
143+
counts the same instructions per row at both sizes, and the rows of
144+
100,000 no longer fit in the caches.
139145

140146
Where Rust is still behind:
141147

142-
- **The biggest views.** Materializing 100,000 rows costs about 1.9 times
143-
as much per row as 10,000 do, as Swift's did here (2.0 times; JS's 1.5
144-
times). What is left is spread over building the view's entries (its
145-
list copies them as it grows), comparing rows to place them, and
146-
dropping them.
147148
- **The smallest changes.** Putting a row in its table's layout compares
148149
its column names with the layout's, so a poke of one row went from 11.4
149150
to 12.4 µs; `pk = N`, an optimistic update, and a round trip are 3–4%

0 commit comments

Comments
 (0)