Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Nereids) offset do more than once when have shuffle after limit (#42576) #42583

Merged
merged 1 commit into from
Oct 29, 2024

Conversation

morrySnow
Copy link
Contributor

pick from master #42576

intro by #39316. it want to fix a problem intro by #36699. but forgot to remove all wrong code in #36699.

after #39316, we should not set offset on exchange, when the exchange is on the top of a limit with offset.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49152 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42b1970c89570a9d2c2a6202ea626772551759db, data reload: false

------ Round 1 ----------------------------------
q1	17686	4404	4365	4365
q2	2049	155	143	143
q3	10271	1879	1890	1879
q4	10286	1259	1281	1259
q5	8309	3819	3936	3819
q6	229	126	150	126
q7	2033	1614	1597	1597
q8	9291	2730	2703	2703
q9	10335	10121	9997	9997
q10	8638	3519	3479	3479
q11	431	250	243	243
q12	466	302	300	300
q13	18341	3967	4036	3967
q14	349	327	319	319
q15	505	456	463	456
q16	537	453	464	453
q17	1128	959	933	933
q18	7124	6924	6784	6784
q19	1673	1577	1568	1568
q20	500	316	298	298
q21	4334	4068	4078	4068
q22	491	396	412	396
Total cold run time: 115006 ms
Total hot run time: 49152 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4296	4270	4301	4270
q2	333	227	225	225
q3	4146	4134	4151	4134
q4	2745	2727	2730	2727
q5	7139	7154	7114	7114
q6	236	121	117	117
q7	3248	2813	2830	2813
q8	4350	4457	4477	4457
q9	14207	13842	13902	13842
q10	4253	4262	4312	4262
q11	738	683	680	680
q12	1039	869	876	869
q13	7623	3735	3795	3735
q14	457	431	434	431
q15	497	465	464	464
q16	634	600	604	600
q17	3822	3803	3783	3783
q18	8875	8749	8783	8749
q19	1711	1670	1667	1667
q20	2373	2142	2131	2131
q21	8649	8719	8829	8719
q22	1079	992	1067	992
Total cold run time: 82450 ms
Total hot run time: 76781 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212209 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42b1970c89570a9d2c2a6202ea626772551759db, data reload: false

query1	3001	466	391	391
query2	9478	2258	2092	2092
query3	6989	196	202	196
query4	23208	21891	21339	21339
query5	19732	6538	6559	6538
query6	415	223	231	223
query7	5280	291	301	291
query8	280	286	255	255
query9	3050	2603	2553	2553
query10	474	312	297	297
query11	15961	14865	14841	14841
query12	126	71	72	71
query13	1000	445	433	433
query14	19230	13529	13349	13349
query15	362	226	230	226
query16	6435	277	258	258
query17	1543	953	936	936
query18	896	315	314	314
query19	214	152	147	147
query20	100	100	95	95
query21	189	100	100	100
query22	5328	5052	5069	5052
query23	34073	33495	33552	33495
query24	6577	6293	6266	6266
query25	539	446	446	446
query26	801	171	160	160
query27	2259	297	292	292
query28	6073	2245	2209	2209
query29	2883	2798	2900	2798
query30	248	164	164	164
query31	965	726	748	726
query32	73	60	62	60
query33	432	249	261	249
query34	864	471	467	467
query35	1149	898	921	898
query36	1509	1178	1130	1130
query37	92	59	63	59
query38	3080	2968	2928	2928
query39	1359	1325	1327	1325
query40	212	94	97	94
query41	40	39	37	37
query42	81	95	89	89
query43	600	546	639	546
query44	1150	716	721	716
query45	244	228	230	228
query46	1227	965	948	948
query47	2022	1678	1932	1678
query48	506	415	422	415
query49	619	372	389	372
query50	850	589	557	557
query51	4807	4703	4787	4703
query52	92	72	89	72
query53	230	182	190	182
query54	2647	2458	2515	2458
query55	88	86	90	86
query56	224	228	210	210
query57	1261	1111	1128	1111
query58	218	221	211	211
query59	3646	3225	3203	3203
query60	218	214	210	210
query61	98	97	103	97
query62	798	468	499	468
query63	205	180	175	175
query64	3211	1563	1537	1537
query65	3631	3507	3572	3507
query66	745	424	368	368
query67	18170	15216	15731	15216
query68	9959	652	625	625
query69	519	269	268	268
query70	1750	1602	1498	1498
query71	424	303	317	303
query72	6958	5077	4994	4994
query73	760	317	319	317
query74	6290	5834	5795	5795
query75	5429	3725	3565	3565
query76	5730	1116	1165	1116
query77	992	258	264	258
query78	12740	11618	11953	11618
query79	6779	641	616	616
query80	1320	390	399	390
query81	490	237	239	237
query82	1677	97	103	97
query83	183	134	131	131
query84	261	71	69	69
query85	884	320	334	320
query86	325	319	293	293
query87	3228	3033	3015	3015
query88	4473	2286	2311	2286
query89	386	288	312	288
query90	1904	213	221	213
query91	162	126	128	126
query92	62	51	52	51
query93	5502	563	570	563
query94	660	209	202	202
query95	2037	1961	2060	1961
query96	635	317	322	317
query97	6478	6408	6447	6408
query98	228	196	210	196
query99	3037	910	966	910
Total cold run time: 326369 ms
Total hot run time: 212209 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42b1970c89570a9d2c2a6202ea626772551759db, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.03
query3	0.25	0.05	0.05
query4	1.79	0.06	0.06
query5	0.53	0.52	0.52
query6	1.30	0.61	0.61
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.49	0.48
query10	0.53	0.53	0.53
query11	0.13	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.79	0.78	0.80
query15	0.78	0.75	0.74
query16	0.39	0.38	0.39
query17	1.02	1.02	1.02
query18	0.21	0.26	0.25
query19	1.90	1.83	1.88
query20	0.01	0.01	0.01
query21	15.43	0.55	0.53
query22	1.92	2.48	1.86
query23	16.98	1.11	1.00
query24	5.30	0.78	3.99
query25	0.40	0.14	0.05
query26	0.77	0.16	0.16
query27	0.04	0.05	0.03
query28	5.62	0.74	0.74
query29	12.65	2.30	2.22
query30	0.55	0.51	0.52
query31	2.82	0.38	0.37
query32	3.39	0.49	0.49
query33	3.08	3.08	3.10
query34	15.29	4.83	4.80
query35	4.83	4.87	4.85
query36	1.08	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 101.55 s
Total hot run time: 30.85 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 42b1970c89570a9d2c2a6202ea626772551759db with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49185 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

------ Round 1 ----------------------------------
q1	17707	4351	4336	4336
q2	2074	154	148	148
q3	10431	1851	1899	1851
q4	10325	1273	1295	1273
q5	8508	3951	3915	3915
q6	242	121	123	121
q7	2054	1633	1605	1605
q8	9288	2710	2688	2688
q9	10478	10056	10024	10024
q10	8636	3504	3468	3468
q11	414	247	243	243
q12	472	293	304	293
q13	18360	3924	4057	3924
q14	357	320	326	320
q15	510	462	459	459
q16	528	459	470	459
q17	1136	984	967	967
q18	7299	6745	6763	6745
q19	1697	1563	1541	1541
q20	518	315	309	309
q21	4429	4111	4094	4094
q22	489	402	404	402
Total cold run time: 115952 ms
Total hot run time: 49185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4286	4319	4286
q2	327	226	223	223
q3	4181	4139	4142	4139
q4	2737	2744	2747	2744
q5	7183	7121	7094	7094
q6	237	122	121	121
q7	3167	2806	2778	2778
q8	4372	4464	4468	4464
q9	14202	13978	13965	13965
q10	4227	4250	4254	4250
q11	787	676	705	676
q12	1032	849	839	839
q13	7552	3945	3748	3748
q14	453	418	427	418
q15	514	462	457	457
q16	618	608	583	583
q17	3839	3815	3853	3815
q18	8816	8818	8884	8818
q19	1767	1648	1634	1634
q20	2414	2239	2212	2212
q21	8612	8570	8411	8411
q22	1031	974	955	955
Total cold run time: 82381 ms
Total hot run time: 76630 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211754 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

query1	927	388	416	388
query2	6553	2277	1992	1992
query3	6915	199	201	199
query4	23361	21385	21476	21385
query5	19746	6498	6499	6498
query6	285	219	237	219
query7	4232	311	310	310
query8	255	269	218	218
query9	3089	2658	2611	2611
query10	433	323	301	301
query11	15545	14889	15121	14889
query12	122	80	76	76
query13	1021	446	434	434
query14	17418	13229	13370	13229
query15	391	215	236	215
query16	6479	282	260	260
query17	1787	924	931	924
query18	907	322	303	303
query19	212	150	149	149
query20	102	97	96	96
query21	185	93	96	93
query22	5255	5014	5018	5014
query23	34210	33311	33455	33311
query24	7888	6274	6254	6254
query25	522	413	417	413
query26	1271	164	162	162
query27	2476	301	293	293
query28	6154	2240	2208	2208
query29	2835	2858	2903	2858
query30	240	168	167	167
query31	921	725	778	725
query32	72	61	64	61
query33	457	265	253	253
query34	864	475	477	475
query35	1168	958	936	936
query36	1236	1305	1204	1204
query37	171	65	65	65
query38	3085	2872	2894	2872
query39	1355	1329	1331	1329
query40	307	99	94	94
query41	39	37	37	37
query42	81	83	86	83
query43	616	630	624	624
query44	1210	727	728	727
query45	248	230	231	230
query46	1235	971	948	948
query47	1860	2013	1667	1667
query48	515	416	398	398
query49	660	385	375	375
query50	865	598	611	598
query51	4727	4704	4672	4672
query52	89	84	79	79
query53	226	179	190	179
query54	2639	2491	2474	2474
query55	80	87	86	86
query56	231	206	211	206
query57	1243	1178	1170	1170
query58	229	221	215	215
query59	3608	3189	3315	3189
query60	231	208	221	208
query61	98	97	100	97
query62	797	474	469	469
query63	205	176	178	176
query64	3585	1642	1475	1475
query65	3634	3507	3547	3507
query66	770	401	423	401
query67	15852	16211	15644	15644
query68	9256	632	621	621
query69	497	259	283	259
query70	1627	1316	1294	1294
query71	400	318	315	315
query72	6790	4912	4796	4796
query73	755	320	318	318
query74	6322	5830	5750	5750
query75	4709	3714	3742	3714
query76	4847	1119	1128	1119
query77	685	256	253	253
query78	12719	11623	11679	11623
query79	7630	656	655	655
query80	1622	387	388	387
query81	500	240	236	236
query82	1632	96	99	96
query83	162	130	137	130
query84	262	69	71	69
query85	882	316	318	316
query86	329	291	283	283
query87	3233	3014	3010	3010
query88	4595	2303	2302	2302
query89	429	291	280	280
query90	1929	207	215	207
query91	162	131	135	131
query92	55	53	50	50
query93	6326	566	526	526
query94	755	214	214	214
query95	2080	1993	1608	1608
query96	656	326	321	321
query97	6606	6398	6446	6398
query98	238	222	192	192
query99	2988	875	864	864
Total cold run time: 317703 ms
Total hot run time: 211754 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.80	0.06	0.06
query5	0.53	0.53	0.53
query6	1.23	0.67	0.61
query7	0.02	0.00	0.01
query8	0.04	0.02	0.02
query9	0.52	0.49	0.49
query10	0.55	0.52	0.53
query11	0.12	0.08	0.09
query12	0.11	0.09	0.08
query13	0.63	0.61	0.61
query14	0.80	0.78	0.79
query15	0.76	0.76	0.76
query16	0.36	0.38	0.36
query17	1.00	1.00	1.05
query18	0.21	0.28	0.23
query19	1.90	1.84	1.82
query20	0.02	0.01	0.01
query21	15.45	0.54	0.53
query22	2.29	2.25	1.51
query23	17.14	1.03	0.93
query24	6.28	0.86	0.90
query25	0.40	0.05	0.05
query26	0.71	0.16	0.15
query27	0.04	0.04	0.03
query28	7.06	0.72	0.70
query29	12.75	2.08	2.27
query30	0.57	0.52	0.53
query31	2.80	0.38	0.36
query32	3.38	0.48	0.49
query33	3.05	3.06	3.06
query34	15.26	4.79	4.82
query35	4.87	4.85	4.87
query36	1.06	1.02	1.02
query37	0.07	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.46 s
Total hot run time: 30.24 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49278 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

------ Round 1 ----------------------------------
q1	17589	4475	4354	4354
q2	2083	158	144	144
q3	10461	1888	1958	1888
q4	10322	1282	1327	1282
q5	8603	3936	3975	3936
q6	233	121	122	121
q7	2043	1637	1581	1581
q8	9295	2811	2724	2724
q9	10617	10007	9998	9998
q10	8671	3535	3544	3535
q11	423	244	250	244
q12	470	290	304	290
q13	18342	3986	4048	3986
q14	352	327	317	317
q15	508	458	478	458
q16	562	443	454	443
q17	1143	983	932	932
q18	7403	6766	6937	6766
q19	1687	1565	1521	1521
q20	539	316	302	302
q21	4427	4084	4071	4071
q22	499	385	412	385
Total cold run time: 116272 ms
Total hot run time: 49278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4302	4307	4280	4280
q2	320	224	223	223
q3	4186	4175	4147	4147
q4	2738	2751	2728	2728
q5	7180	7198	7126	7126
q6	239	120	119	119
q7	3264	2894	2906	2894
q8	4380	4482	4497	4482
q9	14213	13882	13932	13882
q10	4234	4252	4218	4218
q11	771	688	681	681
q12	1037	851	853	851
q13	7346	3765	3734	3734
q14	449	421	434	421
q15	504	467	463	463
q16	623	580	585	580
q17	3812	3889	3768	3768
q18	8819	8679	8784	8679
q19	1724	1676	1683	1676
q20	2365	2137	2084	2084
q21	8564	8475	8513	8475
q22	1046	960	946	946
Total cold run time: 82116 ms
Total hot run time: 76457 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212199 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

query1	935	420	378	378
query2	6534	2272	2218	2218
query3	6925	200	201	200
query4	23022	21367	21490	21367
query5	19725	6557	6522	6522
query6	288	209	240	209
query7	4335	298	303	298
query8	251	258	245	245
query9	3052	2629	2555	2555
query10	424	306	304	304
query11	15647	15378	14909	14909
query12	126	75	71	71
query13	1030	431	426	426
query14	17509	13753	13424	13424
query15	383	218	229	218
query16	6463	282	260	260
query17	1775	933	887	887
query18	896	314	323	314
query19	203	150	145	145
query20	98	96	91	91
query21	195	107	99	99
query22	5259	5194	5102	5102
query23	34041	33431	33665	33431
query24	6938	6268	6291	6268
query25	534	411	439	411
query26	1065	157	159	157
query27	2371	290	291	290
query28	6099	2229	2195	2195
query29	2916	2698	2763	2698
query30	240	166	163	163
query31	981	711	727	711
query32	70	62	55	55
query33	439	260	259	259
query34	858	486	478	478
query35	1149	905	958	905
query36	1328	1102	1035	1035
query37	93	59	60	59
query38	3083	2900	2917	2900
query39	1374	1325	1325	1325
query40	286	93	89	89
query41	40	39	37	37
query42	87	92	99	92
query43	629	587	572	572
query44	1179	718	718	718
query45	243	232	227	227
query46	1240	952	970	952
query47	1841	1821	1750	1750
query48	512	409	397	397
query49	627	374	372	372
query50	854	594	593	593
query51	4743	4660	4638	4638
query52	93	75	82	75
query53	226	181	187	181
query54	2666	2436	2537	2436
query55	82	91	80	80
query56	230	208	205	205
query57	1226	1197	1129	1129
query58	214	201	217	201
query59	3316	3241	3363	3241
query60	225	210	207	207
query61	98	94	95	94
query62	822	577	533	533
query63	196	172	178	172
query64	3551	1584	1498	1498
query65	3657	3513	3558	3513
query66	775	392	404	392
query67	16133	17080	15267	15267
query68	9089	625	636	625
query69	490	284	271	271
query70	1543	1314	1329	1314
query71	394	306	299	299
query72	6943	4794	4814	4794
query73	753	313	315	313
query74	6261	5883	5840	5840
query75	4959	3694	3708	3694
query76	5091	1093	1170	1093
query77	778	249	247	247
query78	12949	11855	12059	11855
query79	5948	602	622	602
query80	1701	381	385	381
query81	474	235	236	235
query82	1545	99	100	99
query83	159	134	132	132
query84	261	70	69	69
query85	983	319	315	315
query86	341	288	313	288
query87	3206	3049	3005	3005
query88	4730	2277	2278	2277
query89	412	273	290	273
query90	1899	214	211	211
query91	159	125	123	123
query92	62	52	50	50
query93	5020	576	561	561
query94	744	207	203	203
query95	2019	2026	2004	2004
query96	644	327	324	324
query97	6563	6369	6553	6369
query98	234	207	202	202
query99	2851	912	888	888
Total cold run time: 313650 ms
Total hot run time: 212199 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.05	0.05
query4	1.78	0.08	0.07
query5	0.53	0.53	0.52
query6	1.23	0.60	0.61
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.53	0.50	0.48
query10	0.53	0.53	0.53
query11	0.12	0.08	0.09
query12	0.12	0.10	0.09
query13	0.61	0.62	0.61
query14	0.79	0.77	0.77
query15	0.78	0.76	0.76
query16	0.38	0.37	0.37
query17	0.99	0.98	1.01
query18	0.22	0.25	0.25
query19	1.94	1.87	1.81
query20	0.01	0.01	0.01
query21	15.46	0.55	0.54
query22	1.95	2.27	1.90
query23	17.40	0.93	0.89
query24	8.35	0.97	0.74
query25	0.36	0.09	0.07
query26	0.87	0.15	0.16
query27	0.05	0.04	0.03
query28	5.24	0.74	0.70
query29	12.60	2.12	2.30
query30	0.58	0.52	0.51
query31	2.82	0.39	0.36
query32	3.38	0.50	0.51
query33	3.06	3.08	3.09
query34	15.30	4.83	4.77
query35	4.86	4.86	4.87
query36	1.06	1.01	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.01	0.01
Total cold run time: 104.7 s
Total hot run time: 30.55 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 469b8e8beda4bcc254176a58403633bb5239ec5c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

…apache#42576)

pick from master apache#42576

intro by apache#39316. it want to fix a problem intro by apache#36699.
but forgot to remove all wrong code in apache#36699.

after apache#39316, we should not set offset on exchange, when the exchange is
on the top of a limit with offset.
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 48685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2569ed455636dd62ba7d7fdb183b4c38194ef42f, data reload: false

------ Round 1 ----------------------------------
q1	17627	4385	4287	4287
q2	2061	151	144	144
q3	10269	1839	1934	1839
q4	10334	1221	1328	1221
q5	8393	3868	3935	3868
q6	239	121	123	121
q7	1997	1614	1618	1614
q8	9319	2722	2937	2722
q9	10199	9846	9701	9701
q10	8634	3500	3496	3496
q11	428	255	236	236
q12	475	304	297	297
q13	18344	3910	4015	3910
q14	357	323	323	323
q15	509	467	464	464
q16	551	453	462	453
q17	1132	933	909	909
q18	7283	6878	6770	6770
q19	1676	1537	1512	1512
q20	542	300	312	300
q21	4395	4128	4109	4109
q22	496	389	403	389
Total cold run time: 115260 ms
Total hot run time: 48685 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4276	4265	4265
q2	329	225	226	225
q3	4125	4118	4140	4118
q4	2743	2747	2739	2739
q5	7242	7155	7132	7132
q6	242	119	121	119
q7	3226	2894	2839	2839
q8	4385	4477	4451	4451
q9	13788	13538	13554	13538
q10	4195	4232	4205	4205
q11	758	671	686	671
q12	1020	856	878	856
q13	7017	3715	3689	3689
q14	449	415	420	415
q15	501	456	455	455
q16	621	602	598	598
q17	3853	3904	3847	3847
q18	8708	8786	8811	8786
q19	1708	1663	1655	1655
q20	2341	2157	2152	2152
q21	8499	8521	8473	8473
q22	1013	923	894	894
Total cold run time: 81070 ms
Total hot run time: 76122 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2569ed455636dd62ba7d7fdb183b4c38194ef42f, data reload: false

query1	927	396	410	396
query2	6547	2246	2107	2107
query3	6927	199	196	196
query4	24125	21756	21598	21598
query5	19738	6475	6524	6475
query6	286	224	239	224
query7	4336	293	316	293
query8	251	259	272	259
query9	3012	2626	2573	2573
query10	467	291	306	291
query11	15534	15035	15434	15035
query12	128	79	74	74
query13	1024	437	429	429
query14	17176	13754	13639	13639
query15	377	231	226	226
query16	6463	280	261	261
query17	1763	892	911	892
query18	892	328	320	320
query19	211	153	149	149
query20	100	100	101	100
query21	188	101	103	101
query22	5195	5064	5002	5002
query23	34395	33539	33638	33539
query24	6922	6293	6297	6293
query25	518	431	420	420
query26	999	160	157	157
query27	2295	299	289	289
query28	6104	2259	2204	2204
query29	2984	2794	2752	2752
query30	245	167	163	163
query31	945	740	739	739
query32	78	62	60	60
query33	442	259	246	246
query34	856	487	489	487
query35	1114	928	915	915
query36	1383	1136	1209	1136
query37	95	60	63	60
query38	3057	2907	2951	2907
query39	1387	1315	1340	1315
query40	207	101	94	94
query41	45	37	37	37
query42	85	93	80	80
query43	569	568	603	568
query44	1186	725	708	708
query45	243	231	229	229
query46	1239	959	947	947
query47	1918	1872	1788	1788
query48	508	415	409	409
query49	630	369	374	369
query50	853	584	599	584
query51	4773	4691	4684	4684
query52	96	86	81	81
query53	228	198	189	189
query54	2638	2449	2491	2449
query55	88	84	85	84
query56	225	198	202	198
query57	1242	1159	1355	1159
query58	222	203	214	203
query59	3572	3347	3208	3208
query60	219	208	205	205
query61	94	92	97	92
query62	797	577	485	485
query63	195	175	171	171
query64	3297	1540	1533	1533
query65	3645	3592	3552	3552
query66	796	404	413	404
query67	15929	15437	17511	15437
query68	9042	625	631	625
query69	500	276	262	262
query70	1672	1442	1443	1442
query71	400	309	312	309
query72	6832	4823	4587	4587
query73	753	319	316	316
query74	6323	5827	5781	5781
query75	4580	3685	3579	3579
query76	4702	1111	1152	1111
query77	588	250	252	250
query78	12509	11520	13046	11520
query79	10674	632	622	622
query80	1123	388	385	385
query81	491	241	236	236
query82	1397	98	92	92
query83	163	130	135	130
query84	260	69	69	69
query85	892	308	313	308
query86	325	322	284	284
query87	3211	2999	2995	2995
query88	4647	2267	2290	2267
query89	476	280	288	280
query90	1961	212	214	212
query91	160	124	124	124
query92	59	50	54	50
query93	6106	579	581	579
query94	780	207	205	205
query95	1998	1960	2113	1960
query96	658	326	342	326
query97	6377	6442	6385	6385
query98	211	207	211	207
query99	2702	822	935	822
Total cold run time: 317597 ms
Total hot run time: 212589 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2569ed455636dd62ba7d7fdb183b4c38194ef42f, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.06	0.05
query4	1.77	0.07	0.07
query5	0.53	0.52	0.53
query6	1.32	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.47
query10	0.53	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.63	0.61	0.62
query14	0.77	0.80	0.78
query15	0.79	0.76	0.75
query16	0.36	0.37	0.37
query17	1.00	1.01	1.02
query18	0.21	0.27	0.24
query19	1.96	1.87	1.82
query20	0.01	0.01	0.01
query21	15.45	0.59	0.58
query22	2.09	1.87	1.57
query23	17.24	0.82	0.96
query24	7.53	1.26	0.92
query25	0.40	0.11	0.05
query26	0.83	0.16	0.16
query27	0.05	0.03	0.03
query28	5.49	0.77	0.70
query29	12.63	2.32	2.17
query30	0.57	0.52	0.50
query31	2.82	0.39	0.37
query32	3.36	0.51	0.48
query33	3.09	3.06	3.08
query34	15.26	4.79	4.77
query35	4.86	4.81	4.86
query36	1.06	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.01	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.14 s
Total hot run time: 30.36 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2569ed455636dd62ba7d7fdb183b4c38194ef42f with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

@morrySnow morrySnow merged commit cbb97f8 into apache:branch-2.0 Oct 29, 2024
21 of 23 checks passed
@morrySnow morrySnow deleted the 2.0_42576 branch October 29, 2024 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants