You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, creating a type transformer requires learning about the TypeTransformer API which requires someone to learn about Flyte Literals, Flyte Context, etc. For example, here is the TypeTransformer for np.ndarray
For the simple case, most data are backed by just a FlyteFile or FlyteDirectory at rest. I propose we have a simplified interface for creating these type transformers. This way, we can make it very easy for users to define their own serializers and deserializers.
Goal: What should the final outcome look like, ideally?
I propose this simplified TypeTransformer API for data backed by FlyteFile:
fromtypingimportProtocol, runtime_checkable, Any, Protocolfromflytekit.types.fileimportFlyteFile@runtime_checkableclassFlyteFileBackedTransformerProtocol(Protocol):
name: strpython_type: typedefsave(self, obj: Any) ->FlyteFile:
"""Serialized obj and return a FlyteFile"""defload(self, file: FlyteFile) ->Any:
"""Load model from file."""
The same API can then be applied to FlyteDirectory.
For example, we can define the NumPy Transformer as:
The proposed simplified TypeTransformer API for data backed by FlyteFile or FlyteDirectory aims to make it easier for users to define their own serializers and deserializers without needing to learn the full TypeTransformer API. This approach would streamline the process by focusing on the essential methods for saving and loading data.
According to the TypeTransformer documentation, creating a custom transformer involves implementing methods like get_literal_type, to_literal, and to_python_value. The proposed API simplifies this by only requiring save and load methods, which handle the serialization to and from FlyteFile.
Would you like more detail on how to implement this simplified API or examples of existing TypeTransformer implementations?
Motivation: Why do you think this is important?
Currently, creating a type transformer requires learning about the TypeTransformer API which requires someone to learn about Flyte Literals, Flyte Context, etc. For example, here is the TypeTransformer for
np.ndarray
For the simple case, most data are backed by just a
FlyteFile
orFlyteDirectory
at rest. I propose we have a simplified interface for creating these type transformers. This way, we can make it very easy for users to define their own serializers and deserializers.Goal: What should the final outcome look like, ideally?
I propose this simplified TypeTransformer API for data backed by FlyteFile:
The same API can then be applied to
FlyteDirectory
.For example, we can define the NumPy Transformer as:
Describe alternatives you've considered
Do nothing and only have
TypeTransformer
API.Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: