-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Copy link
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
When querying a Hive partitioned table, trying to project any of the partition columns after ser/de fails on deserialize. This query, using the test dataset partitioned_table_json fails over Ballista:
select id, part from partitioned_table_jsonTo Reproduce
amazon_reviews is a table Hive partitioned on (marketplace, review_date):
explain select marketplace, review_date, count(*) from amazon_reviews group by marketplace, review_date;On deserialize, column lookup fails here because TableScan schema does not include partition colums:
datafusion/datafusion/proto/src/logical_plan/mod.rs
Lines 382 to 390 in 182d5dc
| let mut projection = None; | |
| if let Some(columns) = &scan.projection { | |
| let column_indices = columns | |
| .columns | |
| .iter() | |
| .map(|name| schema.index_of(name)) | |
| .collect::<Result<Vec<usize>, _>>()?; | |
| projection = Some(column_indices); | |
| } |
Related:
Expected behavior
The plan deserializes without error
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working