Partial sample initialization

[PeakKS]

Brief feature description

It would be nice to have the option to partially initialize a sample. I would create an uninitialized sample based on some maximum capacity of bytes then fill in the data actually received, and then create an initialized sample from those subset of bytes that were actually initialized.

Detailed information

For serialization, it may be expensive to calculate the exact serialized size for something, but much cheaper to figure out a size known to be large enough to contain the serialized data. So an uninitialized sample could be allocated with that cheap function, then the actual serialization can occur, returning the number of bytes actually used, which can then be used to shrink to the final initialized sample.

It would not be necessary for the next sample to reclaim the excess bytes that were not initialized, the overhead should be minimal. The feature would only be about making the initialized sample "correct" without requiring extra metadata of the actual size in a user header.

[elfenpiff]

@PeakKS I think this is already possible with iceoryx2.

If you want to send serialized data and know the size when you loan the sample, you can use the dynamic memory example as a blueprint. In this case, the data type is a slice (a bit like a vector or dynamic array) and when calling loan you can define how many elements you require, see: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/examples/rust/publish_subscribe_dynamic_data/publisher.rs#L48

If you know the worst-case size of your serialization, you can use our static vec

• Rust: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2-bb/container/src/vector/static_vec.rs#L54
• C++: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2-bb/cxx/include/iox2/bb/static_vector.hpp#L35

As data type, or pack it into a struct and send it. The initialization is extremely cheap and fast, even for large vectors (since only the size element is initialized) and then you can grow it dynamically and add elements up to the worst case size. The complex data type examples demonstrates this: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/examples/rust/complex_data_types/complex_data_types.rs

Btw. the C++ and Rust vector are memory layout compatible and you can use them to communicate between C++ and Rust applications without serialization. Just checkout the cross language communication container example:

• Rust: https://github.com/eclipse-iceoryx/iceoryx2/tree/main/examples/rust/cross_language_communication_container
• C++: https://github.com/eclipse-iceoryx/iceoryx2/tree/main/examples/cxx/cross_language_communication_container

Does one of these things address your use case or did you had another API in mind?

[PeakKS]

Yes, I intend to use dynamic slices. Message max extents are based on their content, and there will be multiple message types. But afaict the size of a slice is locked in when the sample is created, without a way to resize it to the final size I want.

Unfortunately have other requirements for communications, so using native iceoryx structs isnt possible.

[elBoberido]

@PeakKS you could guess the size and if it turns out to be not enough, you can loan a new sample with the right amount of memory and copy the data from the old sample to the new one.