Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Add partitioning push down #23432

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Commits on Sep 16, 2024

  1. Configuration menu
    Copy the full SHA
    23cf63b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a6958ef View commit details
    Browse the repository at this point in the history
  3. Optimize ActualProperties compatible checks

    Check boolean nullsAndAnyReplicated field before more complex fields
    dain committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ba97284 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b9b5805 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    0766ed6 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. Configuration menu
    Copy the full SHA
    ae974d9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2321964 View commit details
    Browse the repository at this point in the history
  3. Add partitioning push down and update Hive partitioning

    Add partitioning push down to table scan which connector can use to
    activate optional partitioning, or choose between multiple partitioning
    strategies.
    
    This replaces the existing Metadata makeCompatiblePartitioning and
    getCommonPartitioningHandle methods used exclusively by Hive
    dain committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    5cf85cd View commit details
    Browse the repository at this point in the history
  4. Add partitioning push down to Iceberg

    Add support for pushing plan partitioning into Iceberg when Iceberg
    tables use hash bucked partitioning. This enables co-located joins which
    can be significantly more efficient. Additionally, since Iceberg
    supports multiple independent partitioning functions, a table can
    effectively have multiple distributions, which makes the optimization
    more effective.
    
    This feature can be controlled with the iceberg.bucket-execution-mode
    configuration property and the bucket_execution session property. Mode
    can be set to NEVER, AUTO, or ALWAYS. AUTO is the default and enables
    bucked execution when the bucket count is equal to or greater than the
    current node count.
    dain committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    56f4735 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1ebd691 View commit details
    Browse the repository at this point in the history