-
Notifications
You must be signed in to change notification settings - Fork 168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Alignment Support for Custom Extensions in Encoder/Decoder #245
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #245 +/- ##
==========================================
+ Coverage 98.07% 98.10% +0.02%
==========================================
Files 16 16
Lines 1092 1106 +14
Branches 249 251 +2
==========================================
+ Hits 1071 1085 +14
Misses 21 21 ☔ View full report in Codecov by Sentry. |
Hi, @EddiG. Thank you for your contribution. This PR could boot performance in particular cases, but extending the MessagePack format might break compatibility with other MessagePack processors. Padding is okay, but changing Decoder might not be a good idea. Could you please re-consider the format that does not affect Decoder in order not to break the compatibility? |
Hi @gfx I agree that breaking compatibility is possible in the current state. Let me go ahead and write down my considerations so far. There are two places to add the padding I see.
In the first case, we must be sure that the padding byte doesn't intersect with the other extension type bytes. const decoded = decode(encoded, { extensionCodec, useAlignment: true }); In the second case, we must be sure that we do not mess up the padding and data. So far I see only one way to do that, it is by adding an extra byte that tells us the number of the padding bytes we should offset to reach the actual data. That solution means that we add an extra byte for any data encoded with the extension codec with enabled alignment even if the data is already aligned. What do you think? |
For me, the ideal solution would be to provide the current position in the filling buffer to the I'll give it a try to verify that it is conceptually doable. |
I've proposed another solution here #248 |
Yes, we should conentrait on #248 |
This update introduces a solution to handle data alignment in custom extensions, ensuring that encoded data is aligned in memory according to specified requirements. The main benefit of this approach is enabling zero-copy deserialization, which significantly enhances performance when working with data types like
Float32Array
that have strict alignment needs.Key Steps:
Alignment Handling in Encoding:
Encoder
class, the code checks if an alignment requirement (align
) is specified for the extension before encoding it.0xc1
, known as noop bytes) are inserted before the extension type to ensure the data starts at an aligned memory position.Padding Skipping in Decoding:
Decoder
class is enhanced to recognize and skip padding bytes (0xc1
) when decoding an extension. This ensures the actual extension type is correctly identified, regardless of any alignment padding added during encoding.Storing Alignment Information:
ExtensionCodec
class is modified to store the alignment requirement for each registered extension. This alignment information is utilized during encoding to ensure that data is correctly aligned in memory.Modification of the
ExtData
Class:ExtData
class is extended to include an optionalalign
property. This property stores the alignment requirement for each extension, allowing the encoder to handle data alignment appropriately.