Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][SPARK-49690][SQL] UDT type not showing up in sql type representation in the schema #48174

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ahshahid
Copy link

What changes were proposed in this pull request?

  1. The places where I could see the schema being created from UDT Type, instead of the UDTType, its sql type is used
  2. A Logical Plan containing deserializer is being used to serialize ( i.e show the plan as dataframe), then the target DataType during UpCast, if its of type UDTType, then its sql type is used as target type.
    This is done as a rule in analyzer. I am not sure if that is the best way to handle the issue.

Why are the changes needed?

When showing the schema definition, the UDTType is presented as the ClassName of the UDT Type instead of the sql representation.
For eg: say message field is a UDT with sql representation as
StructField("intField", IntegerType, nullable = false),
StructField("stringField", StringType, nullable = false)))

But the schema contains it as
root
|-- message: test (nullable = true)

instead of
root
|-- message: struct (nullable = true)
| |-- intField: integer (nullable = false)
| |-- stringField: string (nullable = false)

Does this PR introduce any user-facing change?

Yes . The schema containing a UDT type will now be represented using its sql type.

How was this patch tested?

Added a bug test. More tests will be added..

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the SQL label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant