Skip to content

Commit 4a300b0

Browse files
committed
readme: protocol diagrams and bench graphs
Mermaid architecture and pull-sequence diagrams (healthy owner-serves path, dead-owner forced re-ask), plus the 1M bench charts as light/dark SVGs under doc/pull-sharing: convergence at offered rate, per-request cold-vs-warm round-trip, absorption flatness. The latency wording now matches the data: warm hits are flat at every size; the late-sweep cold mean drift is queueing under the sustained bulk pull.
1 parent e8278de commit 4a300b0

7 files changed

Lines changed: 8851 additions & 3 deletions

File tree

‎README.md‎

Lines changed: 69 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,54 @@ one remote store on the SIP path). Pull-sharing does neither:
4343
| SQL ledger (optional) | natural-key `(username, domain, contact)` upsert table: backup (write-back, eager deletes), restart bootstrap with expired-row sweep, lookup of last resort |
4444
| HELD protocol (optional) | `pull_authoritative_serve`: only the owner answers a pull with the value; passive holders answer a compact "held" and are asked directly only when the owner is gone |
4545

46+
```mermaid
47+
flowchart LR
48+
subgraph node1["every cluster node"]
49+
REG["registrar<br/>async lookup"] --> UL["usrloc<br/>pull-sharing mode"]
50+
UL --> CP["cachedb_perf<br/>ul collection + pull"]
51+
end
52+
CP <-- "pulls over bin (clusterer)" --> PEERS[("peer nodes")]
53+
UL -- "write-back upserts, eager deletes" --> DB[("SQL ledger<br/>optional")]
54+
```
55+
56+
## How a pull works
57+
58+
A cold lookup suspends its transaction and asks the cluster; with
59+
`pull_authoritative_serve` on, only the owner ships the bytes:
60+
61+
```mermaid
62+
sequenceDiagram
63+
participant UA as caller
64+
participant N3 as node 3 (cold)
65+
participant N1 as node 1 (owner)
66+
participant N2 as node 2 (passive holder)
67+
UA->>N3: OPTIONS / INVITE alice
68+
Note over N3: miss - transaction suspends,<br/>no worker blocked
69+
N3->>N1: PULL alice
70+
N3->>N2: PULL alice
71+
N1-->>N3: value (authoritative)
72+
N2-->>N3: held
73+
Note over N3: absorb into local memory -<br/>the next lookup is a 0.46 ms local hit
74+
N3->>UA: 302 / routed
75+
```
76+
77+
A held answer proves the record exists, so a dead or restarted-empty
78+
owner costs one extra LAN round trip instead of a failure - even before
79+
the clusterer notices the node is gone:
80+
81+
```mermaid
82+
sequenceDiagram
83+
participant N3 as node 3
84+
participant N1 as node 1 (owner, crashed)
85+
participant N2 as node 2 (passive holder)
86+
N3--xN1: PULL bob
87+
N3->>N2: PULL bob
88+
N2-->>N3: held
89+
Note over N3: no value on offer, but a holder exists:<br/>one forced targeted ask
90+
N3->>N2: PULL bob (force)
91+
N2-->>N3: value (passive copy)
92+
```
93+
4694
## Minimal configuration
4795

4896
```
@@ -93,14 +141,32 @@ Without `db_url` the mode runs as a pure-pull cluster.
93141
* Fill: 1M REGISTERs at 5,000/s aggregate with pulls enabled everywhere —
94142
zero loss, zero livelock (REGISTER never touches the pull machinery).
95143
* Convergence: a cold node absorbed the full 1M set at exactly the offered
96-
lookup rate (3,000/s, 337 s), flat from entry 0 to entry 1M.
144+
lookup rate (3,000/s, 337 s).
97145
* Per-request SIP round-trip: **cold cross-node pull p50 1.4 ms** (p95 9 ms,
98-
p99 37 ms), **warm local hit p50 0.46 ms** (p99 0.78 ms), both independent
99-
of collection size.
146+
p99 37 ms); **warm local hit p50 0.46 ms** (p99 0.78 ms), flat at every
147+
table size — the steady-state cost is size-independent. The cold *mean*
148+
sits near 2 ms for the first ~600k pulls and drifts to ~6 ms late in the
149+
sweep: queueing under the sustained bulk pull, not record-get cost, and
150+
absorption never dropped below the offered rate.
100151
* Tuned run bookkeeping: pulls requested == received == stored == entries ==
101152
1,000,000; timeouts, slot starvation, orphans, misses all zero.
102153
* Warm re-sweep: 1M lookups, 100% answered locally, zero further pulls.
103154

155+
<picture>
156+
<source media="(prefers-color-scheme: dark)" srcset="doc/pull-sharing/convergence-dark.svg">
157+
<img alt="Entries held on the cold node versus time: a straight climb to 1,000,000 at 337 seconds, tracking the offered 3,000/s" src="doc/pull-sharing/convergence-light.svg">
158+
</picture>
159+
160+
<picture>
161+
<source media="(prefers-color-scheme: dark)" srcset="doc/pull-sharing/latency-dark.svg">
162+
<img alt="Per-request round-trip across the sweep: cold pull mean ~2 ms with p95 band, drifting up late in the sweep; warm local hits flat at 0.46 ms" src="doc/pull-sharing/latency-light.svg">
163+
</picture>
164+
165+
<picture>
166+
<source media="(prefers-color-scheme: dark)" srcset="doc/pull-sharing/throughput-dark.svg">
167+
<img alt="Pulls absorbed per second versus entries already held: flat at the offered 3,000/s from zero to one million entries" src="doc/pull-sharing/throughput-light.svg">
168+
</picture>
169+
104170
## Observability
105171

106172
* Exact fleet contact count: `sum(owned_contacts)` across nodes (ownership

0 commit comments

Comments
 (0)