Skip to content

More flexible Parquet encryption configuration #16778

@adamreeve

Description

@adamreeve

The Parquet encryption feature added in #16351 requires specifying AES keys for footer and column encryption directly. This is quite limiting as it assumes all Parquet files in a table use the same keys. It also makes it difficult to implement envelope encryption, where data encryption keys are encrypted with a KMS and the encrypted keys are stored in the Parquet metadata.

DataFusion should provide a more flexible way to configure Parquet encryption and allow file encryption and decryption properties to be generated per-file.

There is some previous discussion about this in #15216

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions