Support GROUP BY GROUPING SETS / ROLLUP / CUBE in the multi-stage query engine#18817
Open
xiangfu0 wants to merge 2 commits into
Open
Support GROUP BY GROUPING SETS / ROLLUP / CUBE in the multi-stage query engine#18817xiangfu0 wants to merge 2 commits into
xiangfu0 wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18817 +/- ##
==========================================
Coverage 64.81% 64.82%
Complexity 1347 1347
==========================================
Files 3396 3398 +2
Lines 212503 212803 +300
Branches 33484 33534 +50
==========================================
+ Hits 137732 137943 +211
- Misses 63610 63679 +69
- Partials 11161 11181 +20
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
commented
Jun 20, 2026
xiangfu0
left a comment
Contributor
Author
There was a problem hiding this comment.
Found one high-signal compatibility issue; see inline comment.
513b62b to
616c781
Compare
xiangfu0
commented
Jun 21, 2026
xiangfu0
left a comment
Contributor
Author
There was a problem hiding this comment.
Found one high-signal issue in the new v2 grouping-sets path; see inline comment.
Contributor
|
Nice contribution! |
b957437 to
600145c
Compare
Jackie-Jiang
reviewed
Jul 1, 2026
e4529c1 to
81145e9
Compare
…lti-stage query engine Adds native execution of GROUP BY GROUPING SETS (...) / ROLLUP(...) / CUBE(...) and the GROUPING() / GROUPING_ID() functions to the multi-stage query engine (MSE), in both the default planner and the v2 physical planner (usePhysicalOptimizer). A grouping-set query is represented as the union of all grouping columns plus, per set, a synthetic $groupingId bitmask (bit i set iff union column i is rolled up). The bitmask is carried as an internal key column that keeps the sets distinct and powers GROUPING() / GROUPING_ID(); the shared conventions live in GroupingSets (pinot-common). Execution paths (identical results on both): - Aggregate directly over a table scan: the whole aggregate is pushed to the single-stage (leaf) engine, which expands each row across the sets and appends $groupingId. - Aggregate over any other input (e.g. after a JOIN): a new RepeatOperator in the MSE runtime performs the same per-set row expansion, then an ordinary GROUP BY over [union keys..., $groupingId] runs -- no grouping-set-specific aggregation logic. Planner split (both planners): LEAF -> EXCHANGE(hash union + $groupingId) -> FINAL(group by union + $groupingId) -> PROJECT. The PROJECT computes GROUPING() / GROUPING_ID() from $groupingId (shared GroupingSetsRexUtils) and drops $groupingId. Guards (explicit failure, never wrong results): at most 31 distinct grouping columns (the $groupingId bitmask is a 32-bit int); WITHIN GROUP ordered aggregates under grouping sets are rejected in the v2 planner (run on single-stage). Tests: GroupingSetsQueriesTest exercises grouping-set aggregation, GROUPING() / GROUPING_ID() (SELECT and HAVING), filtered aggregation, composite/nested sets, explicit GROUPING SETS syntax, grouping sets over a JOIN, and the v2 physical planner -- against both engines via data providers. PlanNodeSerDeTest and RepeatOperatorTest cover the wire format and the runtime row expansion. Rolling upgrade: the feature ships within a single release, so brokers and servers are upgraded together; AggregateNode.groupingSets is additive (proto field 8).
81145e9 to
d1c5838
Compare
…ving the 31-column limit Encode grouping sets the way Calcite does: each set is the list of participating grouping-column indexes (an unbounded per-set membership list) rather than a 32-bit membership bitmask, so the number of grouping columns is no longer capped at 31. A set's position in the plan's set list is its ordinal, carried as the synthetic $groupingId discriminator through the leaf/exchange/final split and the wire. Also addresses review feedback: - RepeatOperator now appends NULLable per-set group-key copies and leaves the original input columns untouched, so aggregation arguments that reference a grouping column (e.g. SUM(b) under ROLLUP(a, b)) aggregate the real values; expansion is streamed one set per block with a termination/usage check. - The per-set expansion is wired in PlanNodeToOpChain (not inside a specific AggregateOperator), so every AggregateOperatorFactory receives already-expanded input and needs no grouping-set awareness. - The GROUPING()/GROUPING_ID() split, projection, LEAF row type, and set encoding are shared between the default and v2 physical planners via GroupingSetsPlanUtils (replaces GroupingSetsRexUtils). - Both planners reject combinations that would otherwise produce broken plans or silently wrong results: WITHIN GROUP / skip-leaf / partitioned-by hints with grouping sets, aggregation-free grouping sets, GROUPING() over a plain GROUP BY, and (v2) grouping sets over an input with no repartitioning exchange. The v2 grouping-set LEAF no longer forwards group-trim collations/limit indexed against the pre-$groupingId layout. - The single-stage engine treats GROUP BY GROUPING SETS (()) (grand-total-only, no grouping columns) as a plain aggregation instead of crashing the reducer. - Wire: plan.proto grouping sets field renumbered to 8; query.thrift/plan.proto rolling-upgrade notes reworded.
d1c5838 to
ffaebbf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds native execution of
GROUP BY GROUPING SETS (...)/ROLLUP(...)/CUBE(...)and theGROUPING()/GROUPING_ID()functions to the multi-stage query engine (MSE), building on thesingle-stage support in #18662.
Two execution paths, both producing identical results:
engine, which expands each row across the grouping sets and appends the synthetic
$groupingIddiscriminator.Reuses the single-stage engine wholesale (the Doris/StarRocks backend-expansion model).
JOIN) — a newRepeatOperatorin the multi-stage runtimeperforms the same per-set row expansion (NULLing rolled-up columns, appending
$groupingId), then an ordinaryGROUP BY over
[union keys…, $groupingId]runs — no grouping-set-specific aggregation logic.Plan split (
PinotAggregateExchangeNodeInsertRule):LEAF → EXCHANGE(hash union+$groupingId) → FINAL(group by union+$groupingId) → PROJECT. The PROJECT computesGROUPING()/GROUPING_ID()from$groupingId(bit extraction), mirroring the single-stage post-aggregation handler, and drops$groupingId.GROUPING/GROUPING_IDare registered inPinotOperatorTable, and explicitGROUP BY GROUPING SETS ((a, b), …)tuple syntax is accepted by the validator.Tests
GroupingSetsQueriesTest— grouping-set aggregation,GROUPING()/GROUPING_ID()(SELECT and HAVING),filtered aggregation, composite/nested sets, explicit GROUPING SETS syntax, and grouping sets over a JOIN —
run against both engines (the order-independent cases via the
useBothQueryEnginesdata provider), plus thesingle-stage genuine-vs-rolled-up-NULL discrimination cases. All green.
Known limitations (rejected with a clear error → never wrong results)
usePhysicalOptimizer=true, opt-in, default off) still rejects grouping sets; thedefault planner is fully supported. (Follow-up.)
$groupingIdbitmask is a 32-bit int) — same cap as thesingle-stage engine.
ORDER BY … LIMITpushdown) is not pushed for grouping-set queries; results are still correct(the broker applies the final
ORDER BY+LIMIT).