Should the library provider a serializer mechanism, and why?

[romm]

This question has regularly been asked, on this repository in #351 and #413, on Twitter and more recently on Mastodon:

“Can Valinor provide a serialization mechanism?”

This library currently provides only one of the two ways that are often used when working with payloads: mapping raw, untyped, unstructured data that can't be trusted, to a strictly typed PHP structure that perfectly fits an application's needs. And this is far from easy, that is why a suite of almost 1500 tests, with a ~98% Mutation Score Indicator is there: to ensure that data that come from anywhere will be mapped correctly, so that developers don't have to worry about it and can focus on business rules and ensure the application runs smoothly.

But there are cases where a structure that was mapped with a payload coming from the outside, also needs to be flattened back to JSON (in most cases I guess) or other plain formats. Although the complexity of this operation is by far easier and can be done natively very easily (for instance by using JsonSerializable::jsonSerialize()), I can relate that writing the same kind of code over and over again can be cumbersome and even error-prone.

This is what this discussion is about, should Valinor be able to do that:

$builder = new MapperBuilder();
$mapper = $builder->mapper();
$serializer = $builder->serializer();

$object = $mapper->map(SomeClass::class, $someJsonSource);
$flattened = $serializer->serialize($object);

$flattened === $someJsonSource; // true, or **ALMOST** (see below)

Things I'm pretty sure about, if Valinor was to provide a serializer:

1. As it was done for the mapper, the serializer mechanism would not leak in the objects. It means that no attribute, interface or trait coming from the library would be used to configure the serializer behaviour. This is an opinionated decision, but this is important to me, and unless someone really succeeds in convincing me, I won't change my mind.

2. The serializer could never guarantee that the result would match the initial output. This is because the mapper can handle custom constructors, and is able to guess which one matches a certain input. See example below:

   final class Color
   {
       public static function fromRGB(int $red, int $green, int $blue): self
       {
           // …
       }
   
       public static function fromHex(string $hex): self
       {
           // …
       }
   }
   
   $mapper = (new MapperBuilder())
       ->registerConstructor(Color::fromRGB(...), Color::fromHex(...))
       ->mapper();
    
   $colorFromRGB = $mapper->map(Color::class, ['red' => 255, 'green' => 0, 'blue' => 0]); // ✅
   $colorFromHex = $mapper->map(Color::class, 'FF0000'); // ✅
   
   // array(3) {'red' => int(255), 'green' => int(0), 'blue' => int(0)}
   $serializer->serialize($colorFromRGB);
   
   // array(3) {'red' => int(255), 'green' => int(0), 'blue' => int(0)}
   // ⚠️ differs from the original input ('FF0000')
   $serializer->serialize($colorFromHex);

   But… is it a problem? The color is still in a valid state, and which format should be used for serialization may still be accessible, for instance with a toRGB() and a toHex() methods.

3. There should be a way for object to customize their serialization behaviour, but I believe it should not come from attributes, rather custom methods inside the objects (not coupled to Valinor) that would be detected and called by the serializer, somehow (needs to be discussed and defined).

4. The serializer would be able to serialize any object, not only those that were mapped by the mapper.

5. Although this would add to the complexity of the library, I think it would not be such a big deal to develop and maintain, because it is far from being as complex as the mapping.

   If some people really see benefits in this feature, and don't want to use the library because it's missing, I might consider starting working on it soon.

---

I would love to hear your thoughts about this and — if you think this is a good idea — how would you like it to work? What are you concerns? etc…

[TimWolla]

As I've said on Mastodon before, I do not think this is necessary / I think this is out of scope. With the limitations you listed, the serialization mechanism would not provide a value-add:

Bullet points (3) and (4) would effectively mean that it is possible for the re-serialized data to not successfully parse again (e.g. serializing to HSL colors when no HSL constructor exists). Round-tripping can only be guaranteed if the library tightly controls both the input and output formats and as Valinor cannot really control those, this expectation which users will have when a serialization feature is provided will easily be violated.

I believe Valinor should focus on providing a great “arbitrary garbage → well-typed object” experience and not distract from that by also adding less-than-great serialization. Right tool for the right job.

---

In case of the “queue” use case that was mentioned in #351, PHP's native serializer supports arbitrary objects graphs perfectly fine. No need for a library with some custom output format if all you need is getting the object back out that you put in within a single application.

[autaut03]

I agree this is out of scope of a mapper - serialization and deserialization are totally different operations with different approaches. At the same time, I do not agree this out of scope of this project as a whole. The reason I believe both must be coupled in a single project is because both share the same context - a node tree describing how a serialized structure maps to native PHP types. They both use the same reflection, the same meta information, the same configuration, the same naming scheme etc. Requiring users to do all of this twice (which are either 2x resource usage or 2x programmers time on development & maintanance) is kind of a waste.

[dantleech]

There should be a way for object to customize their serialization behaviour

Maybe something like:

(new [Mapper|Serializer]Builder())->registerSerialializer(
    Color::class,
    fn (Serializer $_serializer, Color $color) => sprintf('(%d,%d,%d)', $color->red, $color->green, $color->blue), // or whatever
);

Although as @TimWolla points out not being able to define a single round-triippable strategy is a bit of a stumbling block. although it is easy to test round-tripping.

This is certainly something I miss in this library, and I have switched from this package to Symfony Serializer in some cases because I needed both deserialization and serialization.

Is it a good idea? I don't know -- either it can be implemented well or it can't i guess. something like the above could provide a solid "serializer" interface (I think Symfony calls this Normalizing though - converting from object to an "array" from where it's converted to whatever via. an Encoder), does it need to be more than a "Normalizer"? (if it's really handinling "serialization" then I guess the answer is "probably")

[autaut03]

Just a side note - I'm @oprypkhantc.

I have a tiny bit of experience with Java/Kotlin, so I'll be referring to serializers from those languages and comparing Valinor to them. There are three major serializers I've worked with - that is GSON, Moshi, kotlinx.serialization. All are much more mature and bigger than Valinor, so I believe it's valuable to look at them and see what they do good and what they don't :)

This is not to say that everything they do is good; however, all three share a lot of features in common, which are missing from Valinor. That's just something to keep in mind.

I generally agree with all of your bullet points. I'll not be repeating myself saying "I agree" for each of those, but here's a quick rundown of the experience I had in Java and Kotlin. All three of the above-mentioned libraries have a concept of "type adapters". They are implementations responsible for serializing and deserializing a value. Roughly speaking, they are custom constructors from Valinor, but also with serialization. That's how they'd look converted into PHP:

/**
 * @template Type
 */
interface TypeAdapter {
    /**
     * @param Type $value
     */
    public function serialize(mixed $value, Encoder $encoder): void;
    
    /**
     * @return Type
     */
    public function deserialize(Decoder $decoder): mixed;
}

class ColorTypeAdapter implements TypeAdapter {
    public function serialize(mixed $value, Encoder $encoder): void {
        $encoder->encodeString($value->red . ',' . $value->green . ',' . $value->blue);
    }
    
    public function deserialize(Decoder $decoder): mixed {
        $parts = explode(',', $decoder->decodeString());
        
        return new Color($parts[0], $parts[1], $parts[2]);
    }
}

So that's Valinor's custom constructor, plus an additional serialize method. In a rare case where only one of the operations is supported (which generally is not the case), one could simply throw an "unimplemented" exception from the unimplemented method to let the developer know that that's not supported.

All three of the serializers also allow specifying a custom serializer whenever necessary through attributes, this is how that'd look converted to PHP:

class SomeRequest {
    public function __construct() {
        public readonly Color $standard,
        #[SerializeWith(MyNonStandardColorTypeAdapter::class)]
        public readonly Color $nonStandard,
    }
}

or

class SomeRequest {
    public function __construct() {
        public readonly Color $standard,
        #[MyNonStandardColor]
        public readonly Color $nonStandard,
    }
}

Now this is where bullet point #2 comes in. The beauty of this approach (combining both serialize and deserialize into one interface instead of keeping them seperate) is that the user cannot use a different serialization method without also using a different deserialization method, and vice versa, i.e. you cannot do #[SerializeWith(OneSerializer::class)] and then #[DeserializeWith(TwoSerializer::class)].

This way, if a user configures Valinor to use ISO8601DateTimeTypeAdapter for DateTime type, then they're guaranteed to use the same ISO8601 for both serialization and deserialization, under the assumption that ISO8601DateTimeTypeAdapter is tested and does what it's name says. So generally speaking, inconsistencies between serialization and deserialization data formats are not an issue if a user configures the serializer to work that way.

If that's not needed by the user, and they instead want to allow multiple formats on deserialization, then they may do so in the deserialize method of a type adapter. That's what Valinor is doing with multiple custom constructors. Of course, doing so increases the chances of inconsistencies between serailziation and deserialization, but that's user's demand so I also don't see a problem here.

So I believe a single interface is better; it provides a cleaner and simpler to understand concept and it avoids user confusion errors. Especially in a place like serialization, where you generally do not cover your DTO serialization/deserialization with tests as there's really no code to test.
Allowing the user to provide entirely separate implementations for serialization and deserialization is prone to error as the user is more likely to mistankely provide incompatible implementations for serialization and deserialization, wheres with a single interface that's only possible if the implementation itself is bugged.

---

Also a note on attributes. I agree that Valinor should not leak code into objects, i.e. it definitely should not expect userland code to extend anything from Valinor or use specific types from Valinor. However, I do not agree that attributes are part of the code. They were specifically crafted to provide meta information to meta-programming libraries like Valinor and there's value in allowing them. If you completely remove Valinor and leave the attributes, your code would still work as usual, because attributes are completely ignored in runtime unless something specifically uses reflection to instantiate them.

Sure, by default no attributes should be required for Valinor to work. There's enough type information to map objects without any unnecessary attributes. There are cases though where attributes are useful. One of the cases is naming, and it's another pain point in Valinor.

The most common naming scheme for properties/variables in PHP is camelCase and that's part of the code style in many projects. However, that's not the case for JSON APIs - a lot of them actually use snake_case instead. That's not supported by Valinor, meaning you cannot map {"first_name": "Alex"} to class { public string $firstName; } without some sort of source manipulations, but that gets complex way too quickly.

That's why some serializers provide naming strategies and attributes to define/use them in place:

#[NamingStrategy(SnakeCaseNamingStrategy::class)]
class Person {
    public string $firstName;

or

class Person {
    #[SerializedName('first_name')]
    public string $firstName;

What's worse is that there are APIs which incosistenly name the keys, so a single object could contain both snake and camel case entries. That's why a generalized Valinor-instance-level naming strategy will not always work, and why I believe there's a place for attributes.

Of course, naming is not the only valid use case for attributes. Specifying a custom serializer/deserializer for a specific property without changing the property type is also an entirely valid use case, easily solvable with attributes.

[romm]

Thank you for your detailed answers, it is very interesting.

@TimWolla: I believe Valinor should focus on providing a great “arbitrary garbage → well-typed object” experience and not distract from that by also adding less-than-great serialization.

I don't think that adding a serializer would distract me/contributors from providing a powerful mapper. As I stated in my initial post, I really see the mapper as a more complicated and more important matter, a serializer provided alongside would certainly not change that, and require much less maintenance time and energy, at least in the long term.

@TimWolla: In case of the “queue” use case that was mentioned in #351, PHP's native serializer supports arbitrary objects graphs perfectly fine.

There are issues when using PHP's native serializer: the format is handled by PHP only (I think?), meaning the message cannot be easily consumed by an application using a different language. Furthermore, it leaks classes FQCN of the objects directly in the serialized string, which confirms even more my previous point. That's why it is important in this sort of cases (queues & similar) to have a neutral representation of data, often by using JSON.

---

@dantleech: Although as @TimWolla points out not being able to define a single round-triippable strategy is a bit of a stumbling block.

From what I've seen, in other serialization libraries it is never truely guaranteed that back and forth (de)serialization with the exact same data will always be possible, an end user could always abuse it using pure PHP. And as a PHP library author I can tell: if users can, they will.

Take the following example with symfony/serializer for instance:

final readonly class Person
{
    public string $name;
    public int $age;

    public function __construct(string $name, int $age)
    {
        $this->name = strtoupper($name); // ⚠️ Changing the original input
        $this->age = $age;
    }
}

$serializer = new Serializer([new ObjectNormalizer()], [new JsonEncoder()]);

$data = '{"name":"John","age":42}';
$person = $serializer->deserialize($data, Person::class, 'json');
$deserializedData = $serializer->serialize($person, 'json');

$data === $deserializedData; // false ("JOHN" !== "John")

Or with eventsauce/object-hydrator:

$mapper = new ObjectMapperUsingReflection();

$data = ['name' => 'John', 'age' => 42];
$person = $mapper->hydrateObject(Person::class, $data);
$deserializedData = $mapper->serializeObject($person);

$data === $deserializedData; // false ("JOHN" !== "John")

From what I've seen, crell/serde never calls the object constructor, so this is out of this topic's scope.

I've not tested with other libraries.

My point is: in most cases, this won't be a problem because usually data is transferred as-is. When it's not the case, it should be the responsability of the user to explicitly make the conversion — in which case the serializer can and must be configured to “understand” that (for instance by using a method within the object that make the actual conversion).

I really believe that what most users expect from a serializer is the convenience (not having to write a lot of boilerplate code) rather than being sure to retrieve the original input in 100% cases. But maybe I'm wrong, you tell me. 😁

---

@autaut03: However, I do not agree that attributes are part of the code. They were specifically crafted to provide meta information to meta-programming libraries like Valinor and there's value in allowing them.

I've had the night to think about it, and I actually think you're right about it. But I still believe there's a way to achieve the wanted goals without using attributes. I'll do some research work on my side to see if I'm wrong or not.

@autaut03: What's worse is that there are APIs which incosistenly name the keys, so a single object could contain both snake and camel case entries.

This seems to be more of a mapping issue rather than a serializer issue (in that case, Valinor has you covered) or do I get it wrong?

---

I'd be glad to hear if you have more thoughts on this topic, in the meantime I think I'll start a POC to see what can be done, while keeping the primary wanted features in mind:

1. Easily being able to recursively serialize any object/structure with a default behaviour (= reading the properties of objects)
2. Having the possibility to customize how a specific object is serialized
3. Being able to apply general rules, I believe it would mostly be about camelCase => snake_case conversion.

Thanks again for your participation!

[oprypkhantc]

This seems to be more of a mapping issue rather than a serializer issue (in that case, Valinor has you covered) or do I get it wrong?

Well, both. I definitely want naming conversions both ways. Sure, Valinor supports that, but it becomes quite tedious and not error prone if you have many models with custom namings or, even worse, models that are used multiple times under different paths. Having naming attached on a specific property (through an attribute, for example) has it's benefits over what Valinor currently offers: it eliminates duplication of naming and eliminates string property names.

I'd be glad to hear if you have more thoughts on this topic

I naturally would love to see more-or-less all features from GSON or kotlinx.serialization, but that's not realistic. Major things are serialization, naming and userland customization. Besides what's already been pointed out, the only major thing left is userland customization.

What I mean by that is GSON and Moshi allow specifying a factory that accepts a type and returns a type adapter. Valinor almost allow to do that, but it doesn't expose the type intormation and doesn't allow a callable to handle selecting which constructor callback to use for a specific type. Moreover, GSON and Moshi both expose attributes information in that same factory method, allowing the end user to not only customize the way a property is serialized based on it's type, but also on it's meta-information which comes from attributes:

interface TypeAdapterFactory {
    public function create(Type $type, Attributes $attributes, Valinor $valinor): TypeAdapter;
}

TypeAdapterFactory::create() here would be called once for every property, passing in it's type and attributes. That allows the end user to write/create their own attributes and simply check for them (or use information from them) to modify the serialization/deserialization behaviour for that property:

#[Attribute]
class CustomDateTime {
    public function __construct(public readonly string $format) {}
}

class CustomDateTimeTypeAdapterFactory implements TypeAdapterFactory {
    public function create(Type $type, Attributes $attributes, Valinor $valinor): TypeAdapter {
        if (!$attributes->has(CustomDateTime::class)) {
            return $this->next->create($type, $attributes, $valinor);
        }
        
        return new CustomDateTimeTypeAdapter($attributes->one(CustomDateTime::class)->format);
    }
}

class CustomDateTimeTypeAdapter implements TypeAdapter {
    public function __construct(private readonly string $format) {}

    public function serialize($value) { return $value->format($this->format); } 
    public function deserialize($value) { return DateTime::parseFromFormat($value, $this->format); } 
}

Both GSON and Moshi do it exactly this way. Even their internal mappers (for classes, for enums, for primitives) are all implemented using the same interfaces/principles. That means that you can override how all of that is mapped with a single method, which is very powerful.

What's best about this approach is that even if you decide to never use attributes in Valinor itself, it still provides an insane amount of userland customization at a very low maintenance cost, so I though it was worth mentioning. Thank you for reading my lengthy replies :)

[repli2dev]

Through my career I was in companies where the things were simple... on the API you just got payload and mapped it into some DTO hierarchy, performed validations and then accessed the data to get them to the external story. Yet now I am in a company that uses queues processing and that means that we receive the payload through API and map it into DTO hierarchy which we immediatelly after some initial steps put into the queu to pass it into the workers which parse the enriched payload again to perform some operations...

From that perspective:

• Implementing JsonSerializable is tiresome and not error prone, often I see forgotten newly added fields, mismatched formats etc. Possibility to automatically serialize whole object as mentioned is highly welcomed.
• Not providing serialization hinder adaptation as we currently have DTO object serialized by Symfony (De)Normalizer and adaptation would require adding JsonSerializable to everywhere.

1. Not sure if I get the point, but IMHO there should be some serializer settings provided via attributes, such as DateTimeInterface serialization format, or the possibility to skip some field at all.

2. This is fine as far as the serialized content can be deserialized again. Also I think that it is up to developer to maintain that the different constructor methods leads to same value/DTO/... object state.

3. As mentioned in one, there should be some particular specitifaction for simple things like DateTimeInterface... yet I think that for object if it is JsonSerializable than its output should be used... It make sense to provide also a method how developer can regain total control of the serialization process in cases when the output is not that simple.

4. Yes.

I have tried a lots of (de)serialization libraries and it is quite pain to find some usable one as all of those I tried have some quirks and really weird behaviour (such as Symfony Normalizer casting numeric string into stricly typed int/float variable, others not working in PHP 8.2)... So there is still room on the market to rock also in this functionality.

[romm]

Hey! I've opened #423 and would love to have some feedback there; I hope you'll like it. 😊

[romm]

Hey there, I've just released version 1.8.0 which contains the new normalization mechanism. Documentation has been updated as well.

You can now enjoy it in your apps!