Skip to content

Parquet metadata pruning is disabled for scans with a column mapping (Iceberg) #29380

Description

@dancsi

prune_parquet_metadata trims each retained footer down to the projected and predicate columns, shrinking the metadata that Polars Cloud serializes to workers. It matches columns by name.

In Iceberg, columns are identified by field ID, which is stable across renames. If a column mapping exists, we don't do the pruning, because this means that there were column renames.

This popped up as a followup in #29295

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-io-icebergRelated to Apache Iceberg tables.A-io-parquetArea: reading/writing Parquet files

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions