WIP: using arrow-avro. remove own implementation #17861

getChan · 2025-10-01T14:10:36Z

Which issue does this PR close?

Closes Use arrow-avro for performance and improved type support #14097.

todo list

Review whether we can remove public APIs from our own implementations.
Applying arrow-avro 56.0.0 release.
more test

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

remove AvroError
Avro timestamp-*(UTC) types that used to be read with None timezone are now read as UTC. The None timezone applies to the local-timestamp-* types.
remove some pub APIs under /avro_to_arrow/*

alamb · 2025-10-02T10:10:04Z

❤️ amazing! Thank you @getChan
FYI @jecsand838 and @nathaniel-d-ef

alamb · 2025-10-19T20:04:52Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

getChan · 2025-10-19T23:10:10Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

No, not yet. Thanks for release.

nathaniel-d-ef · 2025-10-20T15:21:27Z

Thanks for jumping on this @getChan; let me know if I can help!

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

alamb · 2025-10-29T11:08:03Z

FYI I merged the arrow 57 upgrade to DataFusion -- so if you rebase this PR against main you'll have access to the new arrow-avro crate

# Conflicts: # Cargo.lock

alamb · 2025-12-01T18:46:39Z

datafusion/sqllogictest/test_files/repartition_scan.slt

 ----
 logical_plan TableScan: avro_table projection=[f1, f2, f3]
-physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/testing/data/avro/simple_enum.avro]]}, projection=[f1, f2, f3], file_type=avro
+physical_plan DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/testing/data/avro/simple_enum.avro:0..103], [WORKSPACE_ROOT/testing/data/avro/simple_enum.avro:103..206], [WORKSPACE_ROOT/testing/data/avro/simple_enum.avro:206..309], [WORKSPACE_ROOT/testing/data/avro/simple_enum.avro:309..411]]}, projection=[f1, f2, f3], file_type=avro


this is pretty neat

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/datasource-avro/src/avro_to_arrow/arrow_array_reader.rs # datafusion/datasource-avro/src/avro_to_arrow/schema.rs # datafusion/datasource-avro/src/source.rs

# Conflicts: # Cargo.lock

getChan · 2025-12-20T07:38:50Z

There is a compatibility issue with projection. I'm waiting for a release of arrow-avro that includes the necessary projection features.

DataFusion does not infer the schema from the file when the table schema is explicitly defined.
arrow-avro requires reading the Avro file metadata (avro.schema) to perform projection.
Consequently, projection is problematic when reading Avro tables with explicitly defined schemas.
Please let me know if my understanding is incorrect or if there is a workaround.

alamb · 2025-12-20T14:50:08Z

Thanks for the update @getChan

If the fix already included in arrow-avro (and you are waiting on a release), you could rebase this PR against this branch #19355 to get access to the pre-release code

We would have to wait for the arrow release to actually merge it but it could potentially help unblock your work

I actually would love to get some validation that we can cut over to the new arrow-avro reader before we make the next arrow release (so we can fix any issue that might be found)

jecsand838 · 2025-12-21T03:37:27Z

Thanks for the update @getChan

If the fix already included in arrow-avro (and you are waiting on a release), you could rebase this PR against this branch #19355 to get access to the pre-release code

We would have to wait for the arrow release to actually merge it but it could potentially help unblock your work

I actually would love to get some validation that we can cut over to the new arrow-avro reader before we make the next arrow release (so we can fix any issue that might be found)

@alamb I'm going to start working on apache/arrow-rs#8923 early next week and should have a PR up before Jan 1st.

jecsand838 · 2026-01-13T17:22:46Z

@getChan @alamb My apologies for the delay, but here's the PR which adds projection to the arrow-avro ReaderBuilder API.

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/core/src/datasource/file_format/avro.rs # datafusion/datasource-avro/src/avro_to_arrow/schema.rs # datafusion/datasource-avro/src/source.rs

This reverts commit d62b9d3.

getChan · 2026-01-18T17:57:56Z

Hello. @alamb.
Thanks to the work by jecsand838, I think we’ll be able to apply the code in the next arrow-avro release.
However, as I mentioned in the post above, API change is planned. Do we need any additional steps to prepare for this?

alamb · 2026-01-19T15:11:49Z

Hello. @alamb. Thanks to the work by jecsand838, I think we’ll be able to apply the code in the next arrow-avro release. However, as I mentioned in the post above, API change is planned. Do we need any additional steps to prepare for this?

This is great news @getChan -- thank you! Very exciting.

What API change are you referring to?

If you mean API changes for any users of DataFusion, then I think the best thing to do is the same as for any breaking public API change. It should documented in the upgrading.md guide:
https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading.md

draft: using arrow-avro

e27e1f8

github-actions bot added common Related to common crate datasource Changes to the datasource crate labels Oct 1, 2025

draft: using arrow-avro

027e79d

getChan added 3 commits October 19, 2025 17:15

Merge remote-tracking branch 'origin/main' into arrow-avro

2bf2c89

cargo fmt

2c7e364

check public API's can be replaced

7c9f976

getChan added 3 commits October 27, 2025 23:52

revert arrow dependency

7bc24ea

Merge remote-tracking branch 'origin/main' into arrow-avro

185ede5

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

apply arrow-avro 57.0.0

51557fe

github-actions bot removed the common Related to common crate label Oct 27, 2025

timsaucer mentioned this pull request Oct 28, 2025

Switch from deprecated xz2 to libzma #18331

Closed

getChan added 2 commits October 30, 2025 00:36

BufReader

8e12f47

remove apache-avro dependencies

83ec01d

github-actions bot added core Core DataFusion crate common Related to common crate proto Related to proto crate labels Oct 29, 2025

getChan added 8 commits October 30, 2025 01:16

Merge remote-tracking branch 'origin/main' into arrow-avro

f37fcd3

# Conflicts: # Cargo.lock

remove apache-avro dependencies

b12f563

remove apache-avro dependencies

581ecae

fix test result

74792ff

pass projected schema to arrow-avro

151ea95

testing submodule checkout

78df065

taplo format

c7b3b63

replace with arrow-avro

2556591

getChan added 2 commits November 26, 2025 22:25

remove empty file

c6cebbd

projection indices

3e2667a

jecsand838 mentioned this pull request Nov 27, 2025

Avro schema parser uses type name instead of field name [AVRO] apache/arrow-rs#8928

Open

alamb reviewed Dec 1, 2025

View reviewed changes

getChan added 8 commits December 3, 2025 02:00

check schema should be inferred

d574fe7

fix test table column type

d647ec6

Merge remote-tracking branch 'origin/main' into arrow-avro

5832449

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/datasource-avro/src/avro_to_arrow/arrow_array_reader.rs # datafusion/datasource-avro/src/avro_to_arrow/schema.rs # datafusion/datasource-avro/src/source.rs

resolve conflicts

f6d80fa

resolve conflicts

9404bd6

resolve conflicts

c6b9a7c

use default repartition

c40ecb8

Merge remote-tracking branch 'origin/main' into arrow-avro

b91a351

# Conflicts: # Cargo.lock

use default repartition

d8e0828

getChan added 10 commits January 15, 2026 01:15

apply projection feature in arrow-avro

0b632b3

apply projection feature in arrow-avro

bdd3820

apply projection feature in arrow-avro

9da2388

Merge remote-tracking branch 'origin/main' into arrow-avro

31a07ca

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/core/src/datasource/file_format/avro.rs # datafusion/datasource-avro/src/avro_to_arrow/schema.rs # datafusion/datasource-avro/src/source.rs

resolve conflict

ee742ee

add test for datatype

f7d9203

Update testing submodule for timestamp avro file

d62b9d3

Revert "Update testing submodule for timestamp avro file"

bae3eb6

This reverts commit d62b9d3.

add test for datatype

fe11959

submodule

d65d847

jecsand838 mentioned this pull request Jan 23, 2026

AvroError enum for arrow-avro crate apache/arrow-rs#8759

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: using arrow-avro. remove own implementation #17861

WIP: using arrow-avro. remove own implementation #17861

Uh oh!

getChan commented Oct 1, 2025 •

edited

Loading

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

alamb Dec 1, 2025

Uh oh!

getChan commented Dec 20, 2025

Uh oh!

alamb commented Dec 20, 2025

Uh oh!

jecsand838 commented Dec 21, 2025

Uh oh!

jecsand838 commented Jan 13, 2026

Uh oh!

getChan commented Jan 18, 2026

Uh oh!

alamb commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WIP: using arrow-avro. remove own implementation #17861

Are you sure you want to change the base?

WIP: using arrow-avro. remove own implementation #17861

Uh oh!

Conversation

getChan commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

alamb Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

getChan commented Dec 20, 2025

Uh oh!

alamb commented Dec 20, 2025

Uh oh!

jecsand838 commented Dec 21, 2025

Uh oh!

jecsand838 commented Jan 13, 2026

Uh oh!

getChan commented Jan 18, 2026

Uh oh!

alamb commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

getChan commented Oct 1, 2025 •

edited

Loading