feat(PartitionedOutput): Add dictionary encoding support - #2119
Open
xin-zhang2 wants to merge 2 commits into
Open
feat(PartitionedOutput): Add dictionary encoding support#2119xin-zhang2 wants to merge 2 commits into
xin-zhang2 wants to merge 2 commits into
Conversation
ethanyzhang
force-pushed
the
optimized_partitionedoutput
branch
from
June 11, 2026 23:04
cff9eff to
6944d36
Compare
ethanyzhang
force-pushed
the
optimized_partitionedoutput
branch
from
June 11, 2026 23:17
6944d36 to
974bb09
Compare
xin-zhang2
force-pushed
the
optimized_partitionedoutput
branch
from
June 25, 2026 20:22
974bb09 to
ee25fa7
Compare
xin-zhang2
force-pushed
the
optimized_partitionedoutput
branch
2 times, most recently
from
July 14, 2026 10:07
535f26a to
9611748
Compare
xin-zhang2
force-pushed
the
PartitionedOutput-dict
branch
from
July 21, 2026 12:31
c05b0fa to
85592d6
Compare
Member
Author
|
@yingsu00 Can you please help take a look at this PR when you get a chance? Thanks! |
xin-zhang2
force-pushed
the
optimized_partitionedoutput
branch
2 times, most recently
from
July 27, 2026 14:16
765718f to
1912dee
Compare
yingsu00
approved these changes
Jul 28, 2026
yingsu00
left a comment
yingsu00
left a comment
Collaborator
There was a problem hiding this comment.
Approve the last commit. @xin-zhang2 Could you please clean up this PR? It's 35 commits now.
xin-zhang2
force-pushed
the
PartitionedOutput-dict
branch
from
July 29, 2026 10:05
85592d6 to
94939f1
Compare
Member
Author
|
@yingsu00 I rebased the code. Now it only includes the nested row and dictionary commits. Can you please take another look? Thanks! |
yingsu00
reviewed
Aug 20, 2026
| !parentNulls || bits::isBitSet(parentNulls, i); | ||
| const bool rowIsNull = rawNulls && bits::isBitNull(rawNulls, i); | ||
| if (parentLive && !rowIsNull) { | ||
| chunk[count++] = partitionedValues[i]; |
Collaborator
There was a problem hiding this comment.
Calculate count pre-hand instead of checking if it's full for every row. Have you add benchmark and ran them? Did you not observe regressions on flat values?
Collaborator
There was a problem hiding this comment.
Could you please paste the serializer benchmark results for flat values before and after?
Member
Author
There was a problem hiding this comment.
We only write non-null values. Calculating the count beforehand requires an additional pass to read the parentNulls and rawNulls bitmaps, which is not cheap. if (count == numRowsPerChunk) branch is quite predictable so it won't bring in much cost.
Member
Author
There was a problem hiding this comment.
Here is the result for flat vectors. It shows a significant improvement (~2x) after for cases with null perfect 25, 50, 75.
| PartitioningSerializerBenchmark | time/iter(before) | iters/s(after) |
|---|---|---|
| benchmarkFlushFlat(int_1cols_0pct_1parts) | 148.78us | 151.15us |
| benchmarkFlushFlat(int_1cols_25pct_1parts) | 2.16ms | 784.28us |
| benchmarkFlushFlat(int_1cols_50pct_1parts) | 2.39ms | 1.10ms |
| benchmarkFlushFlat(int_1cols_75pct_1parts) | 3.07ms | 1.82ms |
| benchmarkFlushFlat(int_1cols_100pct_1parts) | 13.03ms | 12.93ms |
| benchmarkFlushFlat(int_1cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(int_1cols_25pct_4parts) | 2.18ms | 687.86us |
| benchmarkFlushFlat(int_1cols_50pct_4parts) | 2.55ms | 1.10ms |
| benchmarkFlushFlat(int_1cols_75pct_4parts) | 3.25ms | 1.79ms |
| benchmarkFlushFlat(int_1cols_100pct_4parts) | 12.64ms | 12.85ms |
| benchmarkFlushFlat(int_1cols_0pct_16parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(int_1cols_25pct_16parts) | 2.33ms | 737.44us |
| benchmarkFlushFlat(int_1cols_50pct_16parts) | 2.75ms | 1.24ms |
| benchmarkFlushFlat(int_1cols_75pct_16parts) | 3.27ms | 1.82ms |
| benchmarkFlushFlat(int_1cols_100pct_16parts) | 13.45ms | 13.84ms |
| benchmarkFlushFlat(int_1cols_0pct_64parts) | 119.36us | 119.24us |
| benchmarkFlushFlat(int_1cols_25pct_64parts) | 2.95ms | 973.36us |
| benchmarkFlushFlat(int_1cols_50pct_64parts) | 3.18ms | 1.63ms |
| benchmarkFlushFlat(int_1cols_75pct_64parts) | 3.65ms | 2.37ms |
| benchmarkFlushFlat(int_1cols_100pct_64parts) | 17.28ms | 18.03ms |
| benchmarkFlushFlat(int_1cols_0pct_256parts) | 312.57us | 299.94us |
| benchmarkFlushFlat(int_1cols_25pct_256parts) | 3.39ms | 1.52ms |
| benchmarkFlushFlat(int_1cols_50pct_256parts) | 3.79ms | 2.39ms |
| benchmarkFlushFlat(int_1cols_75pct_256parts) | 4.97ms | 3.79ms |
| benchmarkFlushFlat(int_1cols_100pct_256parts) | 28.22ms | 27.77ms |
| benchmarkFlushFlat(int_1cols_0pct_1024parts) | 810.69us | 813.99us |
| benchmarkFlushFlat(int_1cols_25pct_1024parts) | 5.25ms | 3.70ms |
| benchmarkFlushFlat(int_1cols_50pct_1024parts) | 6.47ms | 5.70ms |
| benchmarkFlushFlat(int_1cols_75pct_1024parts) | 10.66ms | 9.72ms |
| benchmarkFlushFlat(int_1cols_100pct_1024parts) | 72.93ms | 73.64ms |
| benchmarkFlushFlat(int_4cols_0pct_1parts) | 165.57us | 170.19us |
| benchmarkFlushFlat(int_4cols_25pct_1parts) | 2.19ms | 831.44us |
| benchmarkFlushFlat(int_4cols_50pct_1parts) | 2.46ms | 1.17ms |
| benchmarkFlushFlat(int_4cols_75pct_1parts) | 3.14ms | 1.88ms |
| benchmarkFlushFlat(int_4cols_100pct_1parts) | 13.87ms | 13.50ms |
| benchmarkFlushFlat(int_4cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(int_4cols_25pct_4parts) | 2.23ms | 726.49us |
| benchmarkFlushFlat(int_4cols_50pct_4parts) | 2.46ms | 982.11us |
| benchmarkFlushFlat(int_4cols_75pct_4parts) | 3.38ms | 1.96ms |
| benchmarkFlushFlat(int_4cols_100pct_4parts) | 13.07ms | 13.49ms |
| benchmarkFlushFlat(int_4cols_0pct_16parts) | 101.11us | 101.82us |
| benchmarkFlushFlat(int_4cols_25pct_16parts) | 2.23ms | 737.82us |
| benchmarkFlushFlat(int_4cols_50pct_16parts) | 2.69ms | 1.21ms |
| benchmarkFlushFlat(int_4cols_75pct_16parts) | 3.35ms | 1.92ms |
| benchmarkFlushFlat(int_4cols_100pct_16parts) | 13.47ms | 14.38ms |
| benchmarkFlushFlat(int_4cols_0pct_64parts) | 139.32us | 140.53us |
| benchmarkFlushFlat(int_4cols_25pct_64parts) | 2.98ms | 986.78us |
| benchmarkFlushFlat(int_4cols_50pct_64parts) | 3.44ms | 1.66ms |
| benchmarkFlushFlat(int_4cols_75pct_64parts) | 3.69ms | 2.42ms |
| benchmarkFlushFlat(int_4cols_100pct_64parts) | 17.15ms | 18.45ms |
| benchmarkFlushFlat(int_4cols_0pct_256parts) | 340.24us | 353.61us |
| benchmarkFlushFlat(int_4cols_25pct_256parts) | 3.59ms | 1.61ms |
| benchmarkFlushFlat(int_4cols_50pct_256parts) | 4.02ms | 2.48ms |
| benchmarkFlushFlat(int_4cols_75pct_256parts) | 5.06ms | 4.01ms |
| benchmarkFlushFlat(int_4cols_100pct_256parts) | 27.36ms | 29.52ms |
| benchmarkFlushFlat(int_4cols_0pct_1024parts) | 1.02ms | 1.03ms |
| benchmarkFlushFlat(int_4cols_25pct_1024parts) | 5.29ms | 3.83ms |
| benchmarkFlushFlat(int_4cols_50pct_1024parts) | 6.63ms | 5.73ms |
| benchmarkFlushFlat(int_4cols_75pct_1024parts) | 9.92ms | 9.70ms |
| benchmarkFlushFlat(int_4cols_100pct_1024parts) | 66.69ms | 71.44ms |
| benchmarkFlushFlat(int_16cols_0pct_1parts) | 189.94us | 193.78us |
| benchmarkFlushFlat(int_16cols_25pct_1parts) | 2.54ms | 963.78us |
| benchmarkFlushFlat(int_16cols_50pct_1parts) | 2.70ms | 1.31ms |
| benchmarkFlushFlat(int_16cols_75pct_1parts) | 3.14ms | 1.89ms |
| benchmarkFlushFlat(int_16cols_100pct_1parts) | 13.03ms | 12.98ms |
| benchmarkFlushFlat(int_16cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(int_16cols_25pct_4parts) | 2.73ms | 949.40us |
| benchmarkFlushFlat(int_16cols_50pct_4parts) | 2.97ms | 1.33ms |
| benchmarkFlushFlat(int_16cols_75pct_4parts) | 3.54ms | 2.08ms |
| benchmarkFlushFlat(int_16cols_100pct_4parts) | 13.16ms | 12.97ms |
| benchmarkFlushFlat(int_16cols_0pct_16parts) | 134.74us | 133.90us |
| benchmarkFlushFlat(int_16cols_25pct_16parts) | 2.86ms | 938.61us |
| benchmarkFlushFlat(int_16cols_50pct_16parts) | 2.96ms | 1.25ms |
| benchmarkFlushFlat(int_16cols_75pct_16parts) | 3.38ms | 1.90ms |
| benchmarkFlushFlat(int_16cols_100pct_16parts) | 13.54ms | 13.94ms |
| benchmarkFlushFlat(int_16cols_0pct_64parts) | 216.32us | 215.86us |
| benchmarkFlushFlat(int_16cols_25pct_64parts) | 3.60ms | 1.22ms |
| benchmarkFlushFlat(int_16cols_50pct_64parts) | 3.62ms | 1.88ms |
| benchmarkFlushFlat(int_16cols_75pct_64parts) | 3.82ms | 2.51ms |
| benchmarkFlushFlat(int_16cols_100pct_64parts) | 17.26ms | 17.65ms |
| benchmarkFlushFlat(int_16cols_0pct_256parts) | 586.99us | 592.32us |
| benchmarkFlushFlat(int_16cols_25pct_256parts) | 4.38ms | 2.12ms |
| benchmarkFlushFlat(int_16cols_50pct_256parts) | 4.09ms | 2.57ms |
| benchmarkFlushFlat(int_16cols_75pct_256parts) | 5.41ms | 4.22ms |
| benchmarkFlushFlat(int_16cols_100pct_256parts) | 27.08ms | 27.64ms |
| benchmarkFlushFlat(int_16cols_0pct_1024parts) | 1.90ms | 1.90ms |
| benchmarkFlushFlat(int_16cols_25pct_1024parts) | 6.33ms | 4.75ms |
| benchmarkFlushFlat(int_16cols_50pct_1024parts) | 7.57ms | 6.48ms |
| benchmarkFlushFlat(int_16cols_75pct_1024parts) | 9.83ms | 9.36ms |
| benchmarkFlushFlat(int_16cols_100pct_1024parts) | 60.61ms | 61.45ms |
| benchmarkFlushFlat(int_64cols_0pct_1parts) | 217.28us | 219.15us |
| benchmarkFlushFlat(int_64cols_25pct_1parts) | 4.12ms | 1.55ms |
| benchmarkFlushFlat(int_64cols_50pct_1parts) | 3.14ms | 1.51ms |
| benchmarkFlushFlat(int_64cols_75pct_1parts) | 3.16ms | 1.89ms |
| benchmarkFlushFlat(int_64cols_100pct_1parts) | 13.46ms | 13.34ms |
| benchmarkFlushFlat(int_64cols_0pct_4parts) | 129.65us | 129.36us |
| benchmarkFlushFlat(int_64cols_25pct_4parts) | 4.18ms | 1.29ms |
| benchmarkFlushFlat(int_64cols_50pct_4parts) | 3.59ms | 1.47ms |
| benchmarkFlushFlat(int_64cols_75pct_4parts) | 3.30ms | 1.83ms |
| benchmarkFlushFlat(int_64cols_100pct_4parts) | 13.14ms | 13.38ms |
| benchmarkFlushFlat(int_64cols_0pct_16parts) | 215.99us | 213.40us |
| benchmarkFlushFlat(int_64cols_25pct_16parts) | 4.54ms | 1.57ms |
| benchmarkFlushFlat(int_64cols_50pct_16parts) | 3.77ms | 1.70ms |
| benchmarkFlushFlat(int_64cols_75pct_16parts) | 3.58ms | 2.14ms |
| benchmarkFlushFlat(int_64cols_100pct_16parts) | 13.52ms | 13.90ms |
| benchmarkFlushFlat(int_64cols_0pct_64parts) | 474.07us | 477.28us |
| benchmarkFlushFlat(int_64cols_25pct_64parts) | 5.53ms | 2.28ms |
| benchmarkFlushFlat(int_64cols_50pct_64parts) | 4.40ms | 2.22ms |
| benchmarkFlushFlat(int_64cols_75pct_64parts) | 4.16ms | 2.91ms |
| benchmarkFlushFlat(int_64cols_100pct_64parts) | 17.42ms | 18.12ms |
| benchmarkFlushFlat(int_64cols_0pct_256parts) | 1.50ms | 1.51ms |
| benchmarkFlushFlat(int_64cols_25pct_256parts) | 4.23ms | 2.75ms |
| benchmarkFlushFlat(int_64cols_50pct_256parts) | 6.42ms | 4.25ms |
| benchmarkFlushFlat(int_64cols_75pct_256parts) | 6.59ms | 5.24ms |
| benchmarkFlushFlat(int_64cols_100pct_256parts) | 25.41ms | 26.45ms |
| benchmarkFlushFlat(int_64cols_0pct_1024parts) | 5.62ms | 5.66ms |
| benchmarkFlushFlat(int_64cols_25pct_1024parts) | 9.64ms | 8.67ms |
| benchmarkFlushFlat(int_64cols_50pct_1024parts) | 9.38ms | 8.93ms |
| benchmarkFlushFlat(int_64cols_75pct_1024parts) | 12.04ms | 12.08ms |
| benchmarkFlushFlat(int_64cols_100pct_1024parts) | 34.29ms | 35.17ms |
| benchmarkFlushFlat(bigint_1cols_0pct_1parts) | 149.99us | 144.15us |
| benchmarkFlushFlat(bigint_1cols_25pct_1parts) | 1.09ms | 485.24us |
| benchmarkFlushFlat(bigint_1cols_50pct_1parts) | 1.20ms | 639.57us |
| benchmarkFlushFlat(bigint_1cols_75pct_1parts) | 1.59ms | 1.02ms |
| benchmarkFlushFlat(bigint_1cols_100pct_1parts) | 12.99ms | 12.65ms |
| benchmarkFlushFlat(bigint_1cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(bigint_1cols_25pct_4parts) | 1.07ms | 399.49us |
| benchmarkFlushFlat(bigint_1cols_50pct_4parts) | 1.37ms | 678.65us |
| benchmarkFlushFlat(bigint_1cols_75pct_4parts) | 1.66ms | 1.03ms |
| benchmarkFlushFlat(bigint_1cols_100pct_4parts) | 12.66ms | 13.02ms |
| benchmarkFlushFlat(bigint_1cols_0pct_16parts) | 107.69us | 112.36us |
| benchmarkFlushFlat(bigint_1cols_25pct_16parts) | 1.08ms | 457.86us |
| benchmarkFlushFlat(bigint_1cols_50pct_16parts) | 1.29ms | 654.57us |
| benchmarkFlushFlat(bigint_1cols_75pct_16parts) | 1.73ms | 1.06ms |
| benchmarkFlushFlat(bigint_1cols_100pct_16parts) | 13.97ms | 14.19ms |
| benchmarkFlushFlat(bigint_1cols_0pct_64parts) | 104.44us | 104.03us |
| benchmarkFlushFlat(bigint_1cols_25pct_64parts) | 1.44ms | 537.73us |
| benchmarkFlushFlat(bigint_1cols_50pct_64parts) | 1.48ms | 880.74us |
| benchmarkFlushFlat(bigint_1cols_75pct_64parts) | 1.89ms | 1.36ms |
| benchmarkFlushFlat(bigint_1cols_100pct_64parts) | 17.19ms | 17.58ms |
| benchmarkFlushFlat(bigint_1cols_0pct_256parts) | 245.44us | 246.65us |
| benchmarkFlushFlat(bigint_1cols_25pct_256parts) | 1.74ms | 929.24us |
| benchmarkFlushFlat(bigint_1cols_50pct_256parts) | 1.93ms | 1.33ms |
| benchmarkFlushFlat(bigint_1cols_75pct_256parts) | 2.67ms | 2.13ms |
| benchmarkFlushFlat(bigint_1cols_100pct_256parts | 27.64ms | 27.72ms |
| benchmarkFlushFlat(bigint_1cols_0pct_1024parts) | 659.65us | 665.28us |
| benchmarkFlushFlat(bigint_1cols_25pct_1024parts | 2.80ms | 2.11ms |
| benchmarkFlushFlat(bigint_1cols_50pct_1024parts | 3.49ms | 3.01ms |
| benchmarkFlushFlat(bigint_1cols_75pct_1024parts | 5.38ms | 5.11ms |
| benchmarkFlushFlat(bigint_1cols_100pct_1024part | 69.72ms | 68.74ms |
| benchmarkFlushFlat(bigint_4cols_0pct_1parts) | 150.03us | 156.24us |
| benchmarkFlushFlat(bigint_4cols_25pct_1parts) | 1.12ms | 501.57us |
| benchmarkFlushFlat(bigint_4cols_50pct_1parts) | 1.23ms | 653.53us |
| benchmarkFlushFlat(bigint_4cols_75pct_1parts) | 1.62ms | 1.02ms |
| benchmarkFlushFlat(bigint_4cols_100pct_1parts) | 12.96ms | 12.69ms |
| benchmarkFlushFlat(bigint_4cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(bigint_4cols_25pct_4parts) | 1.15ms | 408.53us |
| benchmarkFlushFlat(bigint_4cols_50pct_4parts) | 1.30ms | 690.49us |
| benchmarkFlushFlat(bigint_4cols_75pct_4parts) | 1.63ms | 946.07us |
| benchmarkFlushFlat(bigint_4cols_100pct_4parts) | 12.71ms | 13.31ms |
| benchmarkFlushFlat(bigint_4cols_0pct_16parts) | 123.53us | 124.99us |
| benchmarkFlushFlat(bigint_4cols_25pct_16parts) | 1.21ms | 467.07us |
| benchmarkFlushFlat(bigint_4cols_50pct_16parts) | 1.27ms | 606.61us |
| benchmarkFlushFlat(bigint_4cols_75pct_16parts) | 1.77ms | 1.17ms |
| benchmarkFlushFlat(bigint_4cols_100pct_16parts) | 13.48ms | 13.97ms |
| benchmarkFlushFlat(bigint_4cols_0pct_64parts) | 121.32us | 121.03us |
| benchmarkFlushFlat(bigint_4cols_25pct_64parts) | 1.51ms | 571.49us |
| benchmarkFlushFlat(bigint_4cols_50pct_64parts) | 1.62ms | 925.28us |
| benchmarkFlushFlat(bigint_4cols_75pct_64parts) | 1.91ms | 1.38ms |
| benchmarkFlushFlat(bigint_4cols_100pct_64parts) | 17.19ms | 17.65ms |
| benchmarkFlushFlat(bigint_4cols_0pct_256parts) | 306.82us | 311.90us |
| benchmarkFlushFlat(bigint_4cols_25pct_256parts) | 1.90ms | 1.04ms |
| benchmarkFlushFlat(bigint_4cols_50pct_256parts) | 2.04ms | 1.37ms |
| benchmarkFlushFlat(bigint_4cols_75pct_256parts) | 2.73ms | 2.19ms |
| benchmarkFlushFlat(bigint_4cols_100pct_256parts | 27.32ms | 27.64ms |
| benchmarkFlushFlat(bigint_4cols_0pct_1024parts) | 891.03us | 908.36us |
| benchmarkFlushFlat(bigint_4cols_25pct_1024parts | 3.19ms | 2.47ms |
| benchmarkFlushFlat(bigint_4cols_50pct_1024parts | 3.86ms | 3.37ms |
| benchmarkFlushFlat(bigint_4cols_75pct_1024parts | 5.58ms | 5.32ms |
| benchmarkFlushFlat(bigint_4cols_100pct_1024part | 67.21ms | 66.88ms |
| benchmarkFlushFlat(bigint_16cols_0pct_1parts) | 182.40us | 185.15us |
| benchmarkFlushFlat(bigint_16cols_25pct_1parts) | 1.50ms | 687.90us |
| benchmarkFlushFlat(bigint_16cols_50pct_1parts) | 1.51ms | 814.94us |
| benchmarkFlushFlat(bigint_16cols_75pct_1parts) | 1.81ms | 1.17ms |
| benchmarkFlushFlat(bigint_16cols_100pct_1parts) | 13.02ms | 13.05ms |
| benchmarkFlushFlat(bigint_16cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(bigint_16cols_25pct_4parts) | 1.58ms | 543.11us |
| benchmarkFlushFlat(bigint_16cols_50pct_4parts) | 1.72ms | 798.86us |
| benchmarkFlushFlat(bigint_16cols_75pct_4parts) | 1.87ms | 1.21ms |
| benchmarkFlushFlat(bigint_16cols_100pct_4parts) | 12.72ms | 13.52ms |
| benchmarkFlushFlat(bigint_16cols_0pct_16parts) | 154.24us | 154.69us |
| benchmarkFlushFlat(bigint_16cols_25pct_16parts) | 1.63ms | 668.24us |
| benchmarkFlushFlat(bigint_16cols_50pct_16parts) | 1.75ms | 865.82us |
| benchmarkFlushFlat(bigint_16cols_75pct_16parts) | 1.93ms | 1.37ms |
| benchmarkFlushFlat(bigint_16cols_100pct_16parts | 13.56ms | 13.90ms |
| benchmarkFlushFlat(bigint_16cols_0pct_64parts) | 195.49us | 196.86us |
| benchmarkFlushFlat(bigint_16cols_25pct_64parts) | 2.05ms | 873.90us |
| benchmarkFlushFlat(bigint_16cols_50pct_64parts) | 1.96ms | 1.18ms |
| benchmarkFlushFlat(bigint_16cols_75pct_64parts) | 2.20ms | 1.61ms |
| benchmarkFlushFlat(bigint_16cols_100pct_64parts | 17.18ms | 17.57ms |
| benchmarkFlushFlat(bigint_16cols_0pct_256parts) | 553.86us | 553.57us |
| benchmarkFlushFlat(bigint_16cols_25pct_256parts | 2.50ms | 1.57ms |
| benchmarkFlushFlat(bigint_16cols_50pct_256parts | 2.80ms | 1.93ms |
| benchmarkFlushFlat(bigint_16cols_75pct_256parts | 2.88ms | 2.38ms |
| benchmarkFlushFlat(bigint_16cols_100pct_256part | 26.82ms | 27.51ms |
| benchmarkFlushFlat(bigint_16cols_0pct_1024parts | 1.82ms | 1.83ms |
| benchmarkFlushFlat(bigint_16cols_25pct_1024part | 3.91ms | 3.28ms |
| benchmarkFlushFlat(bigint_16cols_50pct_1024part | 4.57ms | 4.08ms |
| benchmarkFlushFlat(bigint_16cols_75pct_1024part | 6.49ms | 6.24ms |
| benchmarkFlushFlat(bigint_16cols_100pct_1024par | 59.58ms | 60.71ms |
| benchmarkFlushFlat(bigint_64cols_0pct_1parts) | 405.74us | 398.07us |
| benchmarkFlushFlat(bigint_64cols_25pct_1parts) | 2.03ms | 938.03us |
| benchmarkFlushFlat(bigint_64cols_50pct_1parts) | 1.54ms | 848.78us |
| benchmarkFlushFlat(bigint_64cols_75pct_1parts) | 2.10ms | 1.39ms |
| benchmarkFlushFlat(bigint_64cols_100pct_1parts) | 13.44ms | 13.04ms |
| benchmarkFlushFlat(bigint_64cols_0pct_4parts) | 426.36us | 431.03us |
| benchmarkFlushFlat(bigint_64cols_25pct_4parts) | 2.11ms | 747.49us |
| benchmarkFlushFlat(bigint_64cols_50pct_4parts) | 1.63ms | 747.53us |
| benchmarkFlushFlat(bigint_64cols_75pct_4parts) | 2.13ms | 1.27ms |
| benchmarkFlushFlat(bigint_64cols_100pct_4parts) | 13.23ms | 13.23ms |
| benchmarkFlushFlat(bigint_64cols_0pct_16parts) | 348.90us | 351.32us |
| benchmarkFlushFlat(bigint_64cols_25pct_16parts) | 2.10ms | 837.99us |
| benchmarkFlushFlat(bigint_64cols_50pct_16parts) | 1.86ms | 905.24us |
| benchmarkFlushFlat(bigint_64cols_75pct_16parts) | 2.36ms | 1.55ms |
| benchmarkFlushFlat(bigint_64cols_100pct_16parts | 13.51ms | 13.81ms |
| benchmarkFlushFlat(bigint_64cols_0pct_64parts) | 557.78us | 558.24us |
| benchmarkFlushFlat(bigint_64cols_25pct_64parts) | 2.61ms | 1.28ms |
| benchmarkFlushFlat(bigint_64cols_50pct_64parts) | 2.22ms | 1.34ms |
| benchmarkFlushFlat(bigint_64cols_75pct_64parts) | 2.96ms | 2.24ms |
| benchmarkFlushFlat(bigint_64cols_100pct_64parts | 17.46ms | 17.83ms |
| benchmarkFlushFlat(bigint_64cols_0pct_256parts) | 1.62ms | 1.63ms |
| benchmarkFlushFlat(bigint_64cols_25pct_256parts | 4.07ms | 2.88ms |
| benchmarkFlushFlat(bigint_64cols_50pct_256parts | 3.69ms | 2.89ms |
| benchmarkFlushFlat(bigint_64cols_75pct_256parts | 4.76ms | 4.06ms |
| benchmarkFlushFlat(bigint_64cols_100pct_256part | 25.49ms | 25.61ms |
| benchmarkFlushFlat(bigint_64cols_0pct_1024parts | 5.70ms | 5.71ms |
| benchmarkFlushFlat(bigint_64cols_25pct_1024part | 9.56ms | 8.57ms |
| benchmarkFlushFlat(bigint_64cols_50pct_1024part | 9.28ms | 8.70ms |
| benchmarkFlushFlat(bigint_64cols_75pct_1024part | 8.67ms | 8.48ms |
| benchmarkFlushFlat(bigint_64cols_100pct_1024par | 31.97ms | 32.08ms |
| benchmarkFlushFlat(ldec_1cols_0pct_1parts) | 145.36us | 143.24us |
| benchmarkFlushFlat(ldec_1cols_25pct_1parts) | 567.65us | 316.86us |
| benchmarkFlushFlat(ldec_1cols_50pct_1parts) | 619.07us | 396.44us |
| benchmarkFlushFlat(ldec_1cols_75pct_1parts) | 825.90us | 589.94us |
| benchmarkFlushFlat(ldec_1cols_100pct_1parts) | 13.00ms | 12.72ms |
| benchmarkFlushFlat(ldec_1cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(ldec_1cols_25pct_4parts) | 544.74us | 269.65us |
| benchmarkFlushFlat(ldec_1cols_50pct_4parts) | 567.57us | 307.65us |
| benchmarkFlushFlat(ldec_1cols_75pct_4parts) | 820.53us | 551.19us |
| benchmarkFlushFlat(ldec_1cols_100pct_4parts) | 12.63ms | 13.06ms |
| benchmarkFlushFlat(ldec_1cols_0pct_16parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(ldec_1cols_25pct_16parts) | 565.94us | 276.49us |
| benchmarkFlushFlat(ldec_1cols_50pct_16parts) | 699.86us | 411.90us |
| benchmarkFlushFlat(ldec_1cols_75pct_16parts) | 894.69us | 638.19us |
| benchmarkFlushFlat(ldec_1cols_100pct_16parts) | 13.54ms | 13.89ms |
| benchmarkFlushFlat(ldec_1cols_0pct_64parts) | 108.07us | 107.78us |
| benchmarkFlushFlat(ldec_1cols_25pct_64parts) | 711.44us | 369.40us |
| benchmarkFlushFlat(ldec_1cols_50pct_64parts) | 741.49us | 515.82us |
| benchmarkFlushFlat(ldec_1cols_75pct_64parts) | 961.94us | 813.15us |
| benchmarkFlushFlat(ldec_1cols_100pct_64parts) | 17.31ms | 17.64ms |
| benchmarkFlushFlat(ldec_1cols_0pct_256parts) | 234.86us | 236.40us |
| benchmarkFlushFlat(ldec_1cols_25pct_256parts) | 928.49us | 624.61us |
| benchmarkFlushFlat(ldec_1cols_50pct_256parts) | 1.03ms | 820.57us |
| benchmarkFlushFlat(ldec_1cols_75pct_256parts) | 1.42ms | 1.22ms |
| benchmarkFlushFlat(ldec_1cols_100pct_256parts) | 27.61ms | 27.82ms |
| benchmarkFlushFlat(ldec_1cols_0pct_1024parts) | 579.40us | 592.44us |
| benchmarkFlushFlat(ldec_1cols_25pct_1024parts) | 1.59ms | 1.33ms |
| benchmarkFlushFlat(ldec_1cols_50pct_1024parts) | 2.05ms | 1.86ms |
| benchmarkFlushFlat(ldec_1cols_75pct_1024parts) | 3.01ms | 2.92ms |
| benchmarkFlushFlat(ldec_1cols_100pct_1024parts) | 68.25ms | 70.00ms |
| benchmarkFlushFlat(ldec_4cols_0pct_1parts) | 172.28us | 170.36us |
| benchmarkFlushFlat(ldec_4cols_25pct_1parts) | 636.99us | 360.28us |
| benchmarkFlushFlat(ldec_4cols_50pct_1parts) | 670.74us | 431.99us |
| benchmarkFlushFlat(ldec_4cols_75pct_1parts) | 840.65us | 610.69us |
| benchmarkFlushFlat(ldec_4cols_100pct_1parts) | 12.98ms | 12.71ms |
| benchmarkFlushFlat(ldec_4cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(ldec_4cols_25pct_4parts) | 603.24us | 296.86us |
| benchmarkFlushFlat(ldec_4cols_50pct_4parts) | 659.90us | 378.61us |
| benchmarkFlushFlat(ldec_4cols_75pct_4parts) | 888.24us | 583.57us |
| benchmarkFlushFlat(ldec_4cols_100pct_4parts) | 12.63ms | 12.83ms |
| benchmarkFlushFlat(ldec_4cols_0pct_16parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(ldec_4cols_25pct_16parts) | 619.86us | 306.82us |
| benchmarkFlushFlat(ldec_4cols_50pct_16parts) | 692.40us | 385.69us |
| benchmarkFlushFlat(ldec_4cols_75pct_16parts) | 879.36us | 582.49us |
| benchmarkFlushFlat(ldec_4cols_100pct_16parts) | 13.47ms | 14.06ms |
| benchmarkFlushFlat(ldec_4cols_0pct_64parts) | 135.28us | 136.03us |
| benchmarkFlushFlat(ldec_4cols_25pct_64parts) | 809.57us | 434.94us |
| benchmarkFlushFlat(ldec_4cols_50pct_64parts) | 831.24us | 751.53us |
| benchmarkFlushFlat(ldec_4cols_75pct_64parts) | 999.82us | 893.40us |
| benchmarkFlushFlat(ldec_4cols_100pct_64parts) | 17.17ms | 17.52ms |
| benchmarkFlushFlat(ldec_4cols_0pct_256parts) | 303.40us | 305.44us |
| benchmarkFlushFlat(ldec_4cols_25pct_256parts) | 1.09ms | 749.99us |
| benchmarkFlushFlat(ldec_4cols_50pct_256parts) | 1.19ms | 887.24us |
| benchmarkFlushFlat(ldec_4cols_75pct_256parts) | 1.51ms | 1.30ms |
| benchmarkFlushFlat(ldec_4cols_100pct_256parts) | 27.26ms | 28.25ms |
| benchmarkFlushFlat(ldec_4cols_0pct_1024parts) | 838.69us | 843.94us |
| benchmarkFlushFlat(ldec_4cols_25pct_1024parts) | 2.05ms | 1.73ms |
| benchmarkFlushFlat(ldec_4cols_50pct_1024parts) | 2.38ms | 2.18ms |
| benchmarkFlushFlat(ldec_4cols_75pct_1024parts) | 3.37ms | 3.25ms |
| benchmarkFlushFlat(ldec_4cols_100pct_1024parts) | 66.32ms | 67.03ms |
| benchmarkFlushFlat(ldec_16cols_0pct_1parts) | 177.69us | 186.15us |
| benchmarkFlushFlat(ldec_16cols_25pct_1parts) | 1.04ms | 603.74us |
| benchmarkFlushFlat(ldec_16cols_50pct_1parts) | 782.11us | 522.53us |
| benchmarkFlushFlat(ldec_16cols_75pct_1parts) | 1.05ms | 757.78us |
| benchmarkFlushFlat(ldec_16cols_100pct_1parts) | 13.00ms | 13.04ms |
| benchmarkFlushFlat(ldec_16cols_0pct_4parts) | 0.00fs | 0.00fs |
| benchmarkFlushFlat(ldec_16cols_25pct_4parts) | 912.94us | 418.94us |
| benchmarkFlushFlat(ldec_16cols_50pct_4parts) | 779.44us | 431.57us |
| benchmarkFlushFlat(ldec_16cols_75pct_4parts) | 1.05ms | 709.57us |
| benchmarkFlushFlat(ldec_16cols_100pct_4parts) | 12.67ms | 12.92ms |
| benchmarkFlushFlat(ldec_16cols_0pct_16parts) | 107.07us | 107.94us |
| benchmarkFlushFlat(ldec_16cols_25pct_16parts) | 1.03ms | 475.86us |
| benchmarkFlushFlat(ldec_16cols_50pct_16parts) | 884.44us | 496.44us |
| benchmarkFlushFlat(ldec_16cols_75pct_16parts) | 1.12ms | 821.74us |
| benchmarkFlushFlat(ldec_16cols_100pct_16parts) | 13.51ms | 13.89ms |
| benchmarkFlushFlat(ldec_16cols_0pct_64parts) | 190.90us | 195.19us |
| benchmarkFlushFlat(ldec_16cols_25pct_64parts) | 1.29ms | 722.82us |
| benchmarkFlushFlat(ldec_16cols_50pct_64parts) | 1.03ms | 641.65us |
| benchmarkFlushFlat(ldec_16cols_75pct_64parts) | 1.29ms | 1.14ms |
| benchmarkFlushFlat(ldec_16cols_100pct_64parts) | 17.20ms | 17.89ms |
| benchmarkFlushFlat(ldec_16cols_0pct_256parts) | 529.44us | 529.94us |
| benchmarkFlushFlat(ldec_16cols_25pct_256parts) | 1.75ms | 1.28ms |
| benchmarkFlushFlat(ldec_16cols_50pct_256parts) | 1.59ms | 1.29ms |
| benchmarkFlushFlat(ldec_16cols_75pct_256parts) | 2.04ms | 1.85ms |
| benchmarkFlushFlat(ldec_16cols_100pct_256parts) | 27.01ms | 27.03ms |
| benchmarkFlushFlat(ldec_16cols_0pct_1024parts) | 1.72ms | 1.73ms |
| benchmarkFlushFlat(ldec_16cols_25pct_1024parts) | 2.64ms | 2.46ms |
| benchmarkFlushFlat(ldec_16cols_50pct_1024parts) | 3.55ms | 3.32ms |
| benchmarkFlushFlat(ldec_16cols_75pct_1024parts) | 4.10ms | 4.04ms |
| benchmarkFlushFlat(ldec_16cols_100pct_1024parts | 59.04ms | 60.05ms |
| benchmarkFlushFlat(ldec_64cols_0pct_1parts) | 729.65us | 783.78us |
| benchmarkFlushFlat(ldec_64cols_25pct_1parts) | 2.09ms | 1.21ms |
| benchmarkFlushFlat(ldec_64cols_50pct_1parts) | 1.58ms | 1.05ms |
| benchmarkFlushFlat(ldec_64cols_75pct_1parts) | 1.08ms | 791.44us |
| benchmarkFlushFlat(ldec_64cols_100pct_1parts) | 13.41ms | 13.59ms |
| benchmarkFlushFlat(ldec_64cols_0pct_4parts) | 777.24us | 839.28us |
| benchmarkFlushFlat(ldec_64cols_25pct_4parts) | 2.34ms | 1.32ms |
| benchmarkFlushFlat(ldec_64cols_50pct_4parts) | 1.72ms | 1.07ms |
| benchmarkFlushFlat(ldec_64cols_75pct_4parts) | 1.09ms | 742.78us |
| benchmarkFlushFlat(ldec_64cols_100pct_4parts) | 13.01ms | 13.40ms |
| benchmarkFlushFlat(ldec_64cols_0pct_16parts) | 417.32us | 417.78us |
| benchmarkFlushFlat(ldec_64cols_25pct_16parts) | 1.97ms | 956.28us |
| benchmarkFlushFlat(ldec_64cols_50pct_16parts) | 1.71ms | 998.57us |
| benchmarkFlushFlat(ldec_64cols_75pct_16parts) | 1.17ms | 814.44us |
| benchmarkFlushFlat(ldec_64cols_100pct_16parts) | 13.86ms | 13.88ms |
| benchmarkFlushFlat(ldec_64cols_0pct_64parts) | 775.15us | 766.44us |
| benchmarkFlushFlat(ldec_64cols_25pct_64parts) | 2.44ms | 1.42ms |
| benchmarkFlushFlat(ldec_64cols_50pct_64parts) | 2.20ms | 1.45ms |
| benchmarkFlushFlat(ldec_64cols_75pct_64parts) | 1.54ms | 1.25ms |
| benchmarkFlushFlat(ldec_64cols_100pct_64parts) | 17.54ms | 17.84ms |
| benchmarkFlushFlat(ldec_64cols_0pct_256parts) | 1.86ms | 1.89ms |
| benchmarkFlushFlat(ldec_64cols_25pct_256parts) | 3.93ms | 3.02ms |
| benchmarkFlushFlat(ldec_64cols_50pct_256parts) | 3.69ms | 3.03ms |
| benchmarkFlushFlat(ldec_64cols_75pct_256parts) | 3.09ms | 2.79ms |
| benchmarkFlushFlat(ldec_64cols_100pct_256parts) | 25.09ms | 24.77ms |
| benchmarkFlushFlat(ldec_64cols_0pct_1024parts) | 6.14ms | 6.05ms |
| benchmarkFlushFlat(ldec_64cols_25pct_1024parts) | 9.76ms | 9.13ms |
| benchmarkFlushFlat(ldec_64cols_50pct_1024parts) | 9.48ms | 8.84ms |
| benchmarkFlushFlat(ldec_64cols_75pct_1024parts) | 8.64ms | 8.59ms |
| benchmarkFlushFlat(ldec_64cols_100pct_1024parts | 29.47ms | 29.96ms |
`PrestoIterativePartitioningSerializer` can now serialize struct columns, including structs nested inside structs, with nulls at any level. Serializing a struct column does not write into the vector handed to `append()`. Given `ROW(ROW(ROW(INTEGER)))` where the outer struct is null at row 0 and the inner struct is null at row 2, combining the rows the outer struct discards back into the caller's own null bitmap would leave the caller's inner struct reporting two nulls instead of one, so the mask that tells a child which rows survive is built separately: only when a level really drops rows, and in a buffer of its own when both the level and an ancestor drop rows. A struct whose null bitmap is allocated but holds no null does not count as having nulls. Such a bitmap drops no row, so treating it as nullable would reject `ROW(ROW(VARCHAR))` with "Variable-width columns nested under a ROW with nulls are not supported" and push fixed-width leaves off the bulk copy path onto one stream write per value. A struct column can arrive CONSTANT-encoded, which produces a `PartitionedConstantVector`; that is reported as an unsupported encoding rather than relying on a cast that returns null. The ROW block footer reuses the same null-section writer as simple columns instead of repeating the Velox-to-Presto bitmap conversion. Also adds nested-ROW coverage to the benchmarks: the `PartitionedVector` type generator was off by one nesting level, so its one-level case built a plain BIGINT column, and `ExchangeBenchmark` reused a single null window at every level, so no level ever discarded a row its child would otherwise have written. An unused `PartitionedVector` setter and a test that asserted nothing are removed.
xin-zhang2
force-pushed
the
PartitionedOutput-dict
branch
from
August 24, 2026 09:35
d46a70d to
d620f06
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.