Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GOBBLIN-2159] Adding support for partition level copy in Iceberg distcp #4058

Merged
merged 33 commits into from
Oct 23, 2024
Merged

[GOBBLIN-2159] Adding support for partition level copy in Iceberg distcp #4058

merged 33 commits into from
Oct 23, 2024

Commits on Sep 24, 2024

  1. initial changes for iceberg distcp partition copy

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    02ae2fc View commit details
    Browse the repository at this point in the history

  2. added datetime filter predicate with unit tests

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    981357c View commit details
    Browse the repository at this point in the history

  3. changing string.class to object.class

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    7cd9353 View commit details
    Browse the repository at this point in the history

  4. updated replace partition to use serialized data files

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    82d10d3 View commit details
    Browse the repository at this point in the history

  5. some code cleanup

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    c43d3e1 View commit details
    Browse the repository at this point in the history

  6. added unit test

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    0cf7638 View commit details
    Browse the repository at this point in the history

  7. added replace partition unit test

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    63bb9aa View commit details
    Browse the repository at this point in the history

  8. refactored and added more test

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    6e1cf6b View commit details
    Browse the repository at this point in the history

  9. added javadoc

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    065cde3 View commit details
    Browse the repository at this point in the history

  10. removed extra lines

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a13220d View commit details
    Browse the repository at this point in the history

  11. some minor changes

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    e1d812f View commit details
    Browse the repository at this point in the history

  12. added retry and tests for replace partitions commit step

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    4364044 View commit details
    Browse the repository at this point in the history

  13. minor test changes

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    66d81a3 View commit details
    Browse the repository at this point in the history

  14. added metadata validator

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    24b4823 View commit details
    Browse the repository at this point in the history

  15. removed validator class for now

    @Blazer-007
    Blazer-007 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    d8356e1 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. addressed comments and removed some classes for now

    @Blazer-007
    Blazer-007 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    4dcc88b View commit details
    Browse the repository at this point in the history

  2. fixing checkstyle bugs and disabling newly added tests to find root c… 

    …ause of failure
    @Blazer-007
    Blazer-007 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    46bd976 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. addressed pr comments and added few extra logs

    @Blazer-007
    Blazer-007 committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    e1e6f57 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. refactored classes

    @Blazer-007
    Blazer-007 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    b6163ba View commit details
    Browse the repository at this point in the history

  2. removed extra import statements

    @Blazer-007
    Blazer-007 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    6c73a25 View commit details
    Browse the repository at this point in the history

  3. enabled the tests

    @Blazer-007
    Blazer-007 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9c35733 View commit details
    Browse the repository at this point in the history

  4. fixed iceberg table tests

    @Blazer-007
    Blazer-007 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    cdc863a View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. some refactoring

    @Blazer-007
    Blazer-007 committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    1dbe929 View commit details
    Browse the repository at this point in the history

  2. refactored tests as per review comments

    @Blazer-007
    Blazer-007 committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    383ed91 View commit details
    Browse the repository at this point in the history

  3. throw tablenotfoundexception in place of nosuchtableexception

    @Blazer-007
    Blazer-007 committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    942ad8d View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. fixed throwing proper exception

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    6a4cf78 View commit details
    Browse the repository at this point in the history

  2. removed unused imports

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    2adaa8b View commit details
    Browse the repository at this point in the history

  3. replcaed runtime exception with ioexception

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    c948854 View commit details
    Browse the repository at this point in the history

  4. added check to avoid printing same log line

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    a55ee61 View commit details
    Browse the repository at this point in the history

  5. fixed import order

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    1afc37a View commit details
    Browse the repository at this point in the history

  6. added catch for CheckedExceptionFunction.WrappedIOException wrapper

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    bb35070 View commit details
    Browse the repository at this point in the history

  7. fixed compile issue

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    eeb8d25 View commit details
    Browse the repository at this point in the history

  8. removing unwanted logging

    @Blazer-007
    Blazer-007 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    675e8bb View commit details
    Browse the repository at this point in the history