Skip to content

Conversation

@JonatanMartens
Copy link

@JonatanMartens JonatanMartens commented Nov 29, 2025

Rationale for this change

See #36593
In particular this change is convenient when the column names stored in a file are different from the logical names associated with the columns (see deltalake column mapping feature as an example).

What changes are included in this PR?

Adds the rename_columns method to datasets in pyarrow.
This mehod allows a user to rename columns in the data returned from a scan before actually creating a scanner object.

Are these changes tested?

This PR also add a test for the new rename_columns method using an InMemoryDataset.

Are there any user-facing changes?

Adds the rename_columns method to pyarrow datasets.

@github-actions
Copy link

⚠️ GitHub issue #36593 has been automatically assigned in GitHub to PR creator.

Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the examples where bad you just have to add the imports for doctest to work:
>>> import pyarrow.dataset as ds as seen here:

>>> import pyarrow as pa
>>> table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
... 'n_legs': [2, 2, 4, 4, 5, 100],
... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
... "Brittle stars", "Centipede"]})
>>>
>>> import pyarrow.parquet as pq
>>> pq.write_table(table, "dataset_scanner.parquet")
>>> import pyarrow.dataset as ds
>>> dataset = ds.dataset("dataset_scanner.parquet")
Selecting a subset of the columns:
>>> dataset.scanner(columns=["year", "n_legs"]).to_table()
pyarrow.Table
year: int64
n_legs: int64
----
year: [[2020,2022,2021,2022,2019,2021]]
n_legs: [[2,2,4,4,5,100]]

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Dec 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants