perf(fibre): zero copy rows - #1006
Draft
citizen-stig wants to merge 2 commits into
Draft
Conversation
Adds upload_shard_encode: row proofs for a 148-row shard of a 128MB blob, build_upload_shard, and prost serialisation of the UploadShardRequest. build_upload_shard becomes a hidden pub fn so the bench can call it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPtihoV1GJ9VcNkd7oMRUf
Every row sent to a validator was copied three times: RowInclusionProof
owned a Vec cloned out of the matrix, row_proof_to_blob_row cloned it into
the proto struct, and prost copied it into the request buffer. With 100
validators that is ~476 MiB copied twice more than necessary per 128 MiB
blob.
RowMatrix can now be frozen: its buffer becomes a reference-counted
bytes::Bytes and row_bytes() returns zero-copy slices of it. ExtendedData
freezes the matrix once encoding is done and RowInclusionProof.row is a
Bytes. BlobRow.data and BlobShard.rlcs are generated as Bytes, so building
the upload request only bumps refcounts; prost still writes the request
body once. Mutating a frozen matrix converts it back to an owned buffer.
Public API changes: RowInclusionProof.row is Bytes (was Vec<u8>);
RowMatrix gains freeze/is_frozen/row_bytes; celestia_proto BlobRow.data
and BlobShard.rlcs are Bytes. The download path converts decoded rows to
Vec once (they were moved before), which is a 32 KiB copy per row.
criterion, target-cpu=native, Ryzen 9 3950X (baseline = previous commit,
which only adds the bench):
upload_shard_encode/shard_148_rows_128MB 1.21 ms -> 0.229 ms
(row proofs + build_upload_shard + prost encode, 4.7 MiB request)
blob_row_proofs/shard_148_rows/8MB 17.5 us -> 9.6 us
blob_row_proofs/shard_148_rows/1MB 10.7 us -> 9.7 us
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPtihoV1GJ9VcNkd7oMRUf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview