[SPARK-53771][PYTHON][ARROW] Add support for large list type in Arrow conversion #52498
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Yicong-Huang
changed the title
HyukjinKwon
reviewed
Oct 1, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
Outdated
Show resolved
Hide resolved
Yicong-Huang
force-pushed
the
SPARK-53771/feat/add-useLargeListType-config
branch
from
2c746b1 to
3d88a0b
Compare
ueshin
reviewed
Oct 3, 2025
| errorOnDuplicatedFieldNames: Boolean, | ||
| largeVarTypes: Boolean) { | ||
| largeVarTypes: Boolean, | ||
| largeListType: Boolean = false) { |
There was a problem hiding this comment.
Maybe we shouldn't use the default value to avoid unexpected calls?
There was a problem hiding this comment.
I have removed it.
| errorOnDuplicatedFieldNames: Boolean, | ||
| largeVarTypes: Boolean): Array[Byte] = { | ||
| largeVarTypes: Boolean, | ||
| largeListType: Boolean = false): Array[Byte] = { |
| assert(array4.getInt(2) === 8) | ||
|
|
||
| // Verify that the underlying vector is a LargeListVector | ||
| import org.apache.arrow.vector.complex.LargeListVector |
There was a problem hiding this comment.
Move this import to the import group in the header?
There was a problem hiding this comment.
moved to the header!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a new config
spark.sql.execution.arrow.useLargeListTypethat uses theLargeListArrow type for array columns in Arrow-based operations (UDFs, Pandas conversions, etc.). This Arrow type makes Arrow useLargeListVectorinstead of the regularListVector. This config is disabled by default to maintain the current behavior.Why are the changes needed?
ListVectorhas a size limit of 2 GiB for a single array column in a record batch. This is because it uses 4-byte integers to track the offsets of each array value in the vector. During certain operations with large or deeply nested arrays, it is possible to hit this limit. The most affected scenarios include:applyInPandasoperations with array columns, since the entire group is sent as a single RecordBatchThe
LargeListVectortype uses an 8-byte long to track array value offsets, removing the 2 GiB limit per array column.Does this PR introduce any user-facing change?
Yes, adds a new configuration option
spark.sql.execution.arrow.useLargeListTypethat can help users work around what currently results inIndexOutOfBoundsExceptionor segmentation faults when processing large array columns. This exception being raised is a limitation that should suggest using the large list types instead.How was this patch tested?
ArrowUtilsSuitefor schema conversion with large list typesArrowWriterSuitefor writing data withLargeListVectorArrowTestsMixin(test_large_list_type_config) that verifies the config works correctly with Pandas conversionWas this patch authored or co-authored using generative AI tooling?
Yes, tests generated by Copilot.