Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Config for deciding whether to use Iceberg Time type #11174

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Config for deciding whether to use Iceberg Time type #11174

wants to merge 5 commits into from

Commits on Sep 20, 2024

  1. Config for deciding whether to use Iceberg Time type 

    As per the current design:
If Connect Schema is int32 with logical type is Time, then iceberg table type is returned as TimeType.
Spark does not support the Iceberg TimeType https://iceberg.apache.org/docs/1.4.3/spark-writes/#iceberg-type-to-spark-type.
In this PR, We have added the config to decide whether to use Iceberg Time Type or IntegerType.
This config gives the flexibility to the users to choose IntegerType over TimeType without loosing any precision with being able to query via spark.
    @kumarpritam863
    kumarpritam863 authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f2634a8 View commit details
    Browse the repository at this point in the history

  2. Added Date to int conversion in case iceberg time type conversion is … 

    …not desired.
    @kumarpritam863
    kumarpritam863 authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    661ddaf View commit details
    Browse the repository at this point in the history

  3. Returning integer type if iceberg time type conversion is not intended.

    @kumarpritam863
    kumarpritam863 authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    18c0cee View commit details
    Browse the repository at this point in the history

  4. Fixed RecordConverterTest 

    Fixed Record Converter Test to handle whether to use IcebergTimeType or Integer type for validations.
    @kumarpritam863
    kumarpritam863 authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    20a5c86 View commit details
    Browse the repository at this point in the history

  5. Fixed SchemaUtilsTest 

    Fixed SchemaUtils test to handle iceberg time type or integer type.
    @kumarpritam863
    kumarpritam863 authored Sep 20, 2024
    Configuration menu
    Copy the full SHA
    281681c View commit details
    Browse the repository at this point in the history