[storage] Envelope v2 #3853

muhamadazmy · 2025-10-02T08:15:18Z

[storage] Envelope v2

Summary

Separate header, keys, and other envelope related
information from the actual payload of the envelope
Ability to decode envelope header without decoding the full envelope
Support different encoding for the payload (records) (flexbuffers, or bilrost)
Delay decoding of payload until needed

TODO:

Decode v1 envelope into v2
Use Envelope v2 in code
Convert from v2 to v1 (to write)

Stack created with Sapling. Best reviewed with ReviewStack.

muhamadazmy · 2025-10-07T08:25:03Z

@tillrohrmann in case you want to take a look the interesting parts are in the v2.rs file in this PR

AhmedSoliman · 2025-10-08T13:14:42Z

crates/types/src/message.rs

 }
+
+#[derive(Debug, Clone, Copy, bilrost::Message)]
+pub struct MessageIndexRecrod {


typo: Record.

crates/types/src/storage.rs

AhmedSoliman · 2025-10-08T13:29:28Z

crates/types/src/message.rs

+    pub index: MessageIndex,
+}
+
+bilrost_storage_encode_decode!(MessageIndexRecrod);


Perhaps the type should be named after the wal command (i.e. TruncateOutboxRequest) which btw needs to contain the PartitionId.

Agree! As for the partition-id this record message was mapped 1:1 from v1 without much thought about the payload it carries. I think the reason it doesn't have the partition id is that that's part of the envelope Header (destination)

AhmedSoliman · 2025-10-08T13:36:10Z

crates/wal-protocol/src/v2.rs

+    #[bilrost(3)]
+    dedup: Dedup,
+    #[bilrost(4)]
+    keys: Keys,


I wonder if this is actually needed. On the read path, we ship the keys directly from bifrost (one layer up from the envelope). On the write-path, the appender needs the input record to implement HasRecordKeys which could be implemented on a wrapper type used as input to the appender. Essentially making the input type different than output type to save a few bytes on each record.

AhmedSoliman · 2025-10-08T13:37:34Z

crates/wal-protocol/src/v2.rs

+impl WithPartitionKey for Header {
+    fn partition_key(&self) -> PartitionKey {
+        match self.destination {
+            Destination::None => unimplemented!("expect destinationt to be set"),


typo @ destinationt

AhmedSoliman · 2025-10-08T13:38:53Z

crates/wal-protocol/src/v2.rs

+    #[bilrost(1)]
+    source: Source,
+    #[bilrost(2)]
+    destination: Destination,


destination is potentially unnecessary unless you see a clear use-case for it.

AhmedSoliman · 2025-10-08T13:45:50Z

crates/wal-protocol/src/v2.rs

+    {
+        match kind {
+            StorageCodecKind::FlexbuffersSerde => {
+                todo!("implement loading from envelop V1")


Suggested change

todo!("implement loading from envelop V1")

todo!("implement loading from envelope V1")

AhmedSoliman · 2025-10-08T13:54:37Z

crates/wal-protocol/src/v2.rs

+    record! {
+        @name=PurgeJournal,
+        @kind=RecordKind::PurgeJournal,
+        @payload=PurgeInvocationRequest


is it true that it has the exact same payload as PurgeInvocation?

That's what v1 does. Yes.

# Summary - Separate header, keys, and other envelope related information from the actual payload of the envelope - Ability to decode envelope header without decoding the full envelope - Support different encoding for the payload (records) (flexbuffers, or bilrost) - Delay decoding of payload until needed # TODO: - [x] Decode v1 envelope into v2 - [ ] Use Envelope v2 in code - [ ] Convert from v2 to v1 (to write)

AhmedSoliman · 2025-10-16T16:15:24Z

crates/wal-protocol/src/v2.rs

+/// and serialization details required to interpret the payload.
+#[derive(Debug, Clone, bilrost::Message)]
+pub struct Header {
+    #[bilrost(1)]


Any particular reason why those tags are not 1,2,3?

AhmedSoliman · 2025-11-24T08:58:18Z

crates/types/src/storage.rs

+            where
+                Self: Sized,
+            {
+                debug_assert_eq!(kind, $crate::storage::StorageCodecKind::Bilrost);


nit: I'd suggest we make this an assert_eq

This was referenced Oct 2, 2025

Add partition-store schema metadata accessors #3851

Merged

Move Envelope to v1 #3852

Open

muhamadazmy force-pushed the pr3853 branch from d759eb3 to 5ab1615 Compare October 2, 2025 10:35

muhamadazmy mentioned this pull request Oct 2, 2025

[storage] Draft(2) for Envelope V2 #3854

Closed

muhamadazmy force-pushed the pr3853 branch 5 times, most recently from 75eea1e to 7464a02 Compare October 6, 2025 09:29

muhamadazmy requested review from AhmedSoliman and tillrohrmann October 6, 2025 09:42

AhmedSoliman reviewed Oct 8, 2025

View reviewed changes

muhamadazmy force-pushed the pr3853 branch 3 times, most recently from 60e7920 to 2b53068 Compare October 14, 2025 10:26

muhamadazmy requested a review from AhmedSoliman October 14, 2025 10:38

muhamadazmy force-pushed the pr3853 branch from 2b53068 to a03e298 Compare October 14, 2025 11:33

muhamadazmy force-pushed the pr3853 branch 6 times, most recently from 505bbf7 to 8e7cd15 Compare November 19, 2025 13:12

muhamadazmy marked this pull request as ready for review November 19, 2025 13:14

Move Envelope to v1

c4f920e

muhamadazmy changed the title ~~[storage] Draft for Envelope v2~~ [storage] Envelope v2 Nov 19, 2025

muhamadazmy force-pushed the pr3853 branch 2 times, most recently from c7dba21 to fc48bd2 Compare November 21, 2025 11:19

muhamadazmy marked this pull request as draft November 21, 2025 12:01

muhamadazmy force-pushed the pr3853 branch 2 times, most recently from 93592e3 to a5eea7e Compare November 21, 2025 16:11

muhamadazmy force-pushed the pr3853 branch from a5eea7e to 6192333 Compare November 24, 2025 11:05

muhamadazmy mentioned this pull request Nov 24, 2025

Read and use Envelope v2 #4058

Draft

muhamadazmy marked this pull request as ready for review November 24, 2025 11:16

AhmedSoliman reviewed Nov 25, 2025

View reviewed changes

	todo!("implement loading from envelop V1")
	todo!("implement loading from envelope V1")

[storage] Envelope v2 #3853

Are you sure you want to change the base?

[storage] Envelope v2 #3853

Uh oh!

Conversation

muhamadazmy commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TODO:

Uh oh!

muhamadazmy commented Oct 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

muhamadazmy commented Oct 2, 2025 •

edited

Loading