What we saw
A reader whose accounting allowance with a peer is exhausted takes overDraftRefresh (600 ms) per affected request:
// pkg/retrieval/retrieval.go:127
overDraftRefresh = time.Millisecond * 600
// :224
case <-time.After(overDraftRefresh):
// :253
skip.Add(chunkAddr, peer, overDraftRefresh)
Driving light readers to 171% of their free allowance, 21-26% of requests took that back-off while p50 did not move by a millisecond. Measured at 0.607 to 0.612 s on two independent hosts, which is the constant plus the round trip. Reproduced at 5 concurrent readers and again at 125.
Why it is hard to see
Nothing on that path increments a metric. bee_pushsync_overdraft_refresh exists and correctly reads 0 on a node that only retrieves, since it belongs to the push path. We checked the source at v2.8.1 and a live node's /metrics.
So a node throttled by its own accounting is, on every instrument bee publishes, indistinguishable from a node that is idle. We only found it because the stall latency matched the constant exactly. An operator watching averages or medians would see nothing at all, and the whole effect lives above p90.
#5269 is the neighbouring observation from the log side: the same path is noisy enough in logs to want backoff, while being silent in metrics.
Suggestion
A counter on the retrieval overdraft path, ideally with a histogram of time spent in it, mirroring what pushsync already has:
OverDraftRefreshCount prometheus.Counter // incremented at retrieval.go:224
We are happy to open a PR if the shape is agreed.
We would also value a second opinion on whether the 600 ms constant is meant to be an operator-visible tuning point. In our measurements it is the effective ceiling on free retrieval throughput, roughly 1.2 chunks/s per light peer, and it is currently both invisible and not configurable.
Environment
bee v2.8.1, Linux. Light peers with no chain backend, so they can only ever use the free allowance, which is the same position an in-browser reader is in.
What we saw
A reader whose accounting allowance with a peer is exhausted takes
overDraftRefresh(600 ms) per affected request:Driving light readers to 171% of their free allowance, 21-26% of requests took that back-off while p50 did not move by a millisecond. Measured at 0.607 to 0.612 s on two independent hosts, which is the constant plus the round trip. Reproduced at 5 concurrent readers and again at 125.
Why it is hard to see
Nothing on that path increments a metric.
bee_pushsync_overdraft_refreshexists and correctly reads 0 on a node that only retrieves, since it belongs to the push path. We checked the source at v2.8.1 and a live node's/metrics.So a node throttled by its own accounting is, on every instrument bee publishes, indistinguishable from a node that is idle. We only found it because the stall latency matched the constant exactly. An operator watching averages or medians would see nothing at all, and the whole effect lives above p90.
#5269 is the neighbouring observation from the log side: the same path is noisy enough in logs to want backoff, while being silent in metrics.
Suggestion
A counter on the retrieval overdraft path, ideally with a histogram of time spent in it, mirroring what
pushsyncalready has:We are happy to open a PR if the shape is agreed.
We would also value a second opinion on whether the 600 ms constant is meant to be an operator-visible tuning point. In our measurements it is the effective ceiling on free retrieval throughput, roughly 1.2 chunks/s per light peer, and it is currently both invisible and not configurable.
Environment
bee v2.8.1, Linux. Light peers with no chain backend, so they can only ever use the free allowance, which is the same position an in-browser reader is in.