Rate limit restructure (server-side)#933

Open

khetzl wants to merge 100 commits intomasterfrom

rate_limiting_rework

Commits on Jan 20, 2026

Server-side rate limiting restructured into a separate application.
khetzl
committed
Fix Github Workflow Issue Due to Missing Files (#932)

humaite
authored and
khetzl
committed
Modify default max connections and num acceptors (#943)

humaite
authored and
khetzl
committed
Create a step to list available software/libs (#942)

humaite
authored and
khetzl
committed
Introduce footprint-based indexing & syncing

Lev Berman
authored and
khetzl
committed
Reduce SYNC_CHUNKS_TIMEOUT

Lev Berman
authored and
khetzl
committed
Fix bug where failed entropy generation causes crash

Lev Berman
authored and
khetzl
committed
Increase entropy generation timeout

Lev Berman
authored and
khetzl
committed
Start syncing before entropy is generated

Lev Berman
authored and
khetzl
committed
Report unpad_chunk_error with chunk offset, if possible

Lev Berman
authored and
khetzl
committed
Track invalid packing peers; reduce rating

Lev Berman
authored and
khetzl
committed
Start ar_chunk_storage before ar_data_sync

Lev Berman
authored and
khetzl
committed
Start the HTTP last and shutdown first

Lev Berman
authored and
khetzl
committed
chore: simplify imports

JamesPiechota
authored and
khetzl
committed
chore: move e2e tests into src directory

JamesPiechota
authored and
khetzl
committed
test: drop e2e partition size to 2,000,000

JamesPiechota
authored and
khetzl
committed
fix: fix a race condition to can occur when syncing r29 data

JamesPiechota
authored and
khetzl
committed
fix: allow r29 data to be copied between storage modules

JamesPiechota
authored and
khetzl
committed
fix: when copying data between storage modules include straddling chunk

JamesPiechota
authored and
khetzl
committed
Add ar_block:get_consensus_window_size and get_max_tx_anchor_size

Lev Berman
authored and
khetzl
committed
Introduce data_root syncing process

Lev Berman
authored and
khetzl
committed
Route chunks to all modules with appropriate range

Lev Berman
authored and
khetzl
committed
Sync overlap for every module + the chunk crossing the left border

Lev Berman
authored and
khetzl
committed
Allow cross-module replica 2.9 repacking

Lev Berman
authored and
khetzl
committed
test: continuing to fix e2e tests

JamesPiechota
authored and
khetzl
committed
Remove unused ar_storage_module methods

Lev Berman
authored and
khetzl
committed
Route chunks to storage modules where they start

Lev Berman
authored and
khetzl
committed
Make sure all synced footprint chunks are inside range

Lev Berman
authored and
khetzl
committed
test: consider 2.9 modules for expected unpacked_padded ranges

Lev Berman
authored and
khetzl
committed
Fix GET /data_roots handler crash when called with unconfirmed offset

Lev Berman
authored and
khetzl
committed
Simplify footprint syncing partition iteration

Lev Berman
authored and
khetzl
committed
test: print more details on assert_sync_range failure

Lev Berman
authored and
khetzl
committed
Add ./bin/e2e_shell

Lev Berman
authored and
khetzl
committed
Remove 2.9 chunks leaving disk pool from announced sync_record (global_sync_record)

Lev Berman
authored and
khetzl
committed
Only remove 2.9 chunks from announced sync record

Lev Berman
authored and
khetzl
committed
test: fix asserted no entropy borders in ar_sync_pack_mine_tests

Lev Berman
authored and
khetzl
committed
test: fix setup in ar_mining_server_tests after updating partition size

Lev Berman
authored and
khetzl
committed
Send border chunks to both partitions from disk pool

Lev Berman
authored and
khetzl
committed
test: fix e2e tests

Lev Berman
authored and
khetzl
committed
Fix bug where missing chunk proofs are not handled gracefully

Lev Berman
authored and
khetzl
committed
Fix bug where format_message in ar_process_sampler fails on improper lists

Lev Berman
authored and
khetzl
committed
Fix idle unpacking during replica 2.9 syncing for non-replica_2_9 storage modules

Lev Berman
authored and
khetzl
committed
Fix flaky test_fork_checkpoints_not_found

Lev Berman
authored and
khetzl
committed
test: increase ar_coordinated_mining_tests timeouts

Lev Berman
authored and
khetzl
committed
Send blocks one by one in flaky ar_vdf_block_validation_tests:test_fork_refuse_validation

Lev Berman
authored and
khetzl
committed
Rewrite ar_bench_2_9 to benchmark entropy generation with disk I/O

Lev Berman
authored and
khetzl
committed
Remove lib and releases symlinks from build

JamesPiechota
authored and
khetzl
committed
ar_bench_2_9: simulate the write pattern and disk contention of the node

JamesPiechota
authored and
khetzl
committed
ar_bench_2_9: add packing benchmark

JamesPiechota
authored and
khetzl
committed
ar_bench_2_9: add unpack step to the packing benchmark

JamesPiechota
authored and
khetzl
committed
ar_bench_2_9: rename to ar_bench_packing, remove legacy packing benchmark

JamesPiechota
authored and
khetzl
committed
ar_bench_packing: exit with error if we're unable to fill the write cache

JamesPiechota
authored and
khetzl
committed
fix: ensure r29 unpack_sub_chunk uses entropy cache

JamesPiechota
authored and
khetzl
committed
fix: make user entropy cache process is started

JamesPiechota
authored and
khetzl
committed
chore: minor logging changes

JamesPiechota
authored and
khetzl
committed
fix: have metrics log the raw HTTP status code instead of converting it to a string

JamesPiechota
authored and
khetzl
committed
upd: add a metric for entropy cache events

JamesPiechota
authored and
khetzl
committed
upd: limit number of footprints that are synced concurrently

JamesPiechota
authored and
khetzl
committed
refactor: simplifying ar_data_sync_worker_master logic

JamesPiechota
authored and
khetzl
committed
fix: log whenever an HTTP response status is unknown

JamesPiechota
authored and
khetzl
committed
upd: set entropy cache in MiB rather than entropies

JamesPiechota
authored and
khetzl
committed
fix: better http status metrics

JamesPiechota
authored and
khetzl
committed
fix: group sync_range tasks by footprint

JamesPiechota
authored and
khetzl
committed
upd: log error/warning/info/debug to debug logs

JamesPiechota
authored and
khetzl
committed
fix: log config at startup

JamesPiechota
authored and
khetzl
committed
fix: sort footprint key by partition and footprint before peer

JamesPiechota
authored and
khetzl
committed
fix: ar_entropy_cache is no longer a gen_server

JamesPiechota
authored and
khetzl
committed
upd: add replica_2_9_entropy cache metric

JamesPiechota
authored and
khetzl
committed
fix: don't cache entropy when preparing

JamesPiechota
authored and
khetzl
committed
upd: add sync_chunks_skipped metric to track when a chunk is synced but not written

JamesPiechota
authored and
khetzl
committed
fix: don't sync a chunk if it's already recorded

JamesPiechota
authored and
khetzl
committed
Account for same data roots crossing block borders

Lev Berman
authored and
khetzl
committed
Move data root validation to ar_data_root_sync.erl

Lev Berman
authored and
khetzl
committed
Fix data roots validation in multi-tx blocks

Lev Berman
authored and
khetzl
committed
chore: remove some verbose logging

JamesPiechota
authored and
khetzl
committed
fix: cut footprint queue when we cut peer queue, cache active footprint count to avoid constant recalculating

JamesPiechota
authored and
khetzl
committed
chore: rename data sync task states

JamesPiechota
authored and
khetzl
committed
wip: fixing footprint activation bugs

JamesPiechota
authored and
khetzl
committed
refactor: split out peer handling when syncing into ar_peer

JamesPiechota
authored and
khetzl
committed
fix: don't allow peer_workers to have infinte waiting tasks

JamesPiechota
authored and
khetzl
committed
fix: track max footprints in ar_data_sync_coordinator rather than per peer

JamesPiechota
authored and
khetzl
committed
test: add ar_data_sync_coordinator and ar_peer_worker tests

JamesPiechota
authored and
khetzl
committed
fix: purge idle peer workers

JamesPiechota
authored and
khetzl
committed
feat: define some rules for cursor agents

JamesPiechota
authored and
khetzl
committed
test: add e2e test for large storage modules

JamesPiechota
authored and
khetzl
committed
test: fix new large storage module e2e test

JamesPiechota
authored and
khetzl
committed
fix: skip ahead to next entropy partition once we've collected one chunk from each footprint

JamesPiechota
authored and
khetzl
committed
test: introduce ar_test_runner, allow running one test

JamesPiechota
authored and
khetzl
committed
upd: add some metrics to track data discovery

JamesPiechota
authored and
khetzl
committed
fix: update GET /footprints release to 89

JamesPiechota
authored and
khetzl
committed
fix: don't register atoms for each ar_peer_worker process (could exhaust atom table)

JamesPiechota
authored and
khetzl
committed
fix: fix race condition that could cause footprint limit to be exceeded

JamesPiechota
authored and
khetzl
committed
chore: add some cursor rules

JamesPiechota
authored and
khetzl
committed
fix: fix tests and typos

JamesPiechota
authored and
khetzl
committed
fix: recent changes broke the active footprint record keeping

JamesPiechota
authored and
khetzl
committed
fix: increase gen_server:call timeout for ar_peer_worker

JamesPiechota
authored and
khetzl
committed
test: fix ar_data_sync_coordinator tests

JamesPiechota
authored and
khetzl
committed
fix tests again
khetzl
committed
remove accidentally commited emacs temp file
khetzl
committed
Merge branch 'master' into rate_limiting_rework
khetzl
committed