Can I create a custom provider that offers serialization?

[phi-friday]

To apply serialization to polars DataFrame, I created the following library airflow-serde-polars.
However, I didn't find anything about serialization in the provider meta information, so I didn't define it as a provider.

Do you have any plans for this? Or is it just something I didn't find?

[potiuk]

There is quite a bit of a different approaches to serialization in Airlfow and we have this issue #40974 to address it as part of Airflow 3. It really neads someone to take a lead and figure out how to actually consolidate serialization code.

I think - and this appeared in a few discussions with @bolkedebruin before hta tit would be good idea indeed to add "serialization" extendability to providers - so that serde extensions can live in providers rather than in airlfow core.

Might be a good idea to implement it this way by extending provider's manager and adding this capability - otherwise we will keep on accumulating dependencies to core airflow by adding more stuff.

[bolkedebruin]

What @potiuk is referring to is that we aim to merge / consolidate serialization into serde. @gyli is working on that. Given that you are extending serde this is fine, but we now need to find a place where to place the extensions in a way that is future proof.

In that context - as @potiuk mentioned, having this in the providers makes sense. Given that polars isn't a strong dependency of Airflow (not really warranting its own provider I guess) the best place is (imho) to have it in common.serialization or common.serde. Wdyt @gyli ?

[phi-friday]

Based on what you said, my understanding is that

1. work is in progress to integrate serialization functionality.
2. there is no officially supported method until 1st is done.
3. the best I can do at the moment is an unofficially supported method (adding a module to airflow.serialization.serializers).

Am I understanding this correctly?

[potiuk]

Almost.

the best I can do at the moment is an unofficially supported method (adding a module to airflow.serialization.serializers).

Since we released Airlfow 2.10 and we do not plan any more releases in 2.* series except a bridge 2.11 release that will only have new "Airflow 3" migration features, anything you add now as feature to main is going to be released in Airflow 3. So the best you can do is really to work on the "serialization consolidation" workstream with @gyli and - possibly - add a way to extend serializations in providers.

Not sure which provider - but we toyed with "apache airflow providers dataframe" provider - https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l - and I think that would be a great candidate to add such serialization there. But again -it's going to be only available in Airflow 3 unless we decide to backport provider's support for serialization to Airflow 2.11.

[gyli]

I agree with @bolkedebruin and @potiuk that such serialization should live in the providers ideally. serde can be extended to register serializer in provider folders. So @phi-friday , we can have an offline discussion if you plan to implement this feature, to avoid the potential conflicted changes to serde. We can surely split the work here.

I think the scope of this feature does not have too much overlap with "serialization consolidation", and it is not the consolidation that block your development of the new serializer, assuming only a new serializer is needed, which will be modulized. Again, the ability of serde to register serializer under providers should be implemented first.