Skip to content
This repository was archived by the owner on Dec 1, 2025. It is now read-only.
This repository was archived by the owner on Dec 1, 2025. It is now read-only.

Pre-filter columns within reduce for potential dask-expr improvement #92

@dougbrn

Description

@dougbrn

map_partitions is an opaque layer for utilizing dask-expr's optimization features. We frequently use map_partitions so it may be of limited use, but it would be useful to get in front of map_partitions with column pre-filtering whenever we can. reduce is a straightforward first case.

The advantage being that we can potentially avoid loading columns we don't use, though it will absolutely be dependent on the user workflow and where it invokes other map_partitions calls

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions