Lb/sync replica 2 9 #812

ldmberman · 2025-05-22T15:54:38Z

No description provided.

apps/arweave/src/ar_data_sync.erl

apps/arweave/src/ar_http_iface_middleware.erl

apps/arweave/src/ar_http_iface_server.erl

apps/arweave/src/ar_data_discovery.erl

apps/arweave/src/ar_data_sync.erl

apps/arweave/src/ar_footprint_record.erl

apps/arweave/include/ar_data_discovery.hrl

ldmberman · 2025-07-04T11:55:08Z

A summary of the recent changes:

The footprint record is now maintained only for 256 KiB chunks before the strict threshold and all chunks after the threshold.
The footprint record is maintained only for replica 2.9 data.
The “normal” and “footprint” syncing procedures are scheduled in phases: first, the normal procedure completes one iteration over the partition range; then the footprint procedure completes one iteration; then the normal procedure runs again, and so on.
The “footprint” procedure syncs only replica 2.9 data by ignoring non-replica 2.9 packing returned by GET /footprints.
The “normal” procedure syncs only non-replica 2.9 data (replica 2.9 data is already excluded from the normal sync record on master).
Because both procedures are running, a two-phase deployment is no longer necessary.
Footprint records are initialized from the existing sync records on startup.
Added tests for ar_footprint_record and ar_peer_intervals.
Replica 2.9 syncing does not start before the entropy generation has been completed.
The server-side blocking of serving of replica 2.9 chunks (along with the BLOCK_2_9_SYNCING flag) is removed.
Put entropy generation behind a mutex to avoid redundant entropy generation; logging entropy_generation_lock_collision on collision.
Added tracking and reporting of redundant entropy generation (the number of entropy generations per key in the last 30 minutes).

This reverts commit e8ab455.

This reverts commit be3b703.

Since it is primarily a wrapper around access to an ETS table the gen_server layer was causing some race conditions. Now that it's a striaght interface to ETS we can rely on ETS to resolve certain classes of race conditins (e.g. what happened before is a `put` was implemented as a `cast` and before the `cast` was processed a straight read from the ETS table was done and missed the put update)

…ut not written

when syncing data roots

Adding a new application in rebar3 tree requires specific files, at least on arweave point of view. After the release is compiled, a github job will create artifacts (to be reused in other jobs), but an error is produced instead: tar: ./_build/default/lib/arweave_diagnostic/include: File removed before we read it tar: ./_build/default/lib/arweave_diagnostic/priv: File removed before we read it tar: ./_build/default/rel/arweave/lib/arweave_diagnostic-0.0.1/include: File removed before we read it tar: ./_build/default/rel/arweave/lib/arweave_diagnostic-0.0.1/priv: File removed before we read it tar: ./_build/test/lib/arweave_diagnostic/include: File removed before we read it tar: ./_build/test/lib/arweave_diagnostic/priv: File removed before we read it tar: ./_build/test/rel/arweave/lib/arweave_diagnostic-0.0.1/include: File removed before we read it tar: ./_build/test/rel/arweave/lib/arweave_diagnostic-0.0.1/priv: File removed before we read it include and priv folders are required. Those must be present in the github repository as well.

…nt count to avoid constant recalculating

also try to reduce some of the coordinator and peer_worker reductions be replacing gen_server:call with gen_server:cast and caching the formatted ppeer

… peer

…tprints" This reverts commit 19c9dee.