Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](load) disable num segments check in compatibility mode #41053

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

kaijchen
Copy link
Contributor

Proposed changes

When using mixed version of BEs, sink v2 on old BE won't report num segments to load streams on the new BE.
This will cause false positive segment num mismatch.

This PR addressed this issue by disabling num segments check when any tablets_to_commit proto has not set num_segments field.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41662 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0333f8aa163b66b718fbfae983644b3e44e2c9d, data reload: false

------ Round 1 ----------------------------------
q1	17801	7432	7297	7297
q2	2065	164	158	158
q3	10707	1121	1146	1121
q4	10368	770	710	710
q5	7766	3076	3058	3058
q6	242	148	151	148
q7	1018	612	595	595
q8	9421	2066	2072	2066
q9	6819	6399	6467	6399
q10	7015	2247	2335	2247
q11	438	251	246	246
q12	407	224	216	216
q13	17783	2982	3009	2982
q14	236	225	210	210
q15	586	528	507	507
q16	696	625	612	612
q17	971	796	793	793
q18	7445	6770	6816	6770
q19	1481	959	1069	959
q20	586	282	281	281
q21	4155	3286	3320	3286
q22	1123	1001	1037	1001
Total cold run time: 109129 ms
Total hot run time: 41662 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7277	7219	7216	7216
q2	332	229	227	227
q3	3064	3016	2960	2960
q4	2105	1854	1821	1821
q5	5593	5609	5687	5609
q6	236	142	146	142
q7	2216	1775	1785	1775
q8	3316	3448	3431	3431
q9	8780	8906	8746	8746
q10	3445	3436	3475	3436
q11	586	492	493	492
q12	841	665	591	591
q13	9430	3177	3172	3172
q14	300	259	278	259
q15	586	532	534	532
q16	706	688	678	678
q17	1831	1619	1604	1604
q18	8406	7764	7794	7764
q19	1721	1627	1517	1517
q20	2105	1879	1882	1879
q21	5577	5309	5511	5309
q22	1106	1102	1040	1040
Total cold run time: 69559 ms
Total hot run time: 60200 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195622 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0333f8aa163b66b718fbfae983644b3e44e2c9d, data reload: false

query1	1260	868	847	847
query2	6382	2023	1989	1989
query3	10771	4022	4035	4022
query4	62303	27286	23551	23551
query5	5188	484	465	465
query6	406	169	172	169
query7	5426	298	294	294
query8	296	212	209	209
query9	7986	2626	2600	2600
query10	468	297	266	266
query11	17467	15214	15719	15214
query12	165	99	108	99
query13	1419	438	411	411
query14	10826	7363	7706	7363
query15	217	180	179	179
query16	6525	482	462	462
query17	1135	586	574	574
query18	1362	321	327	321
query19	213	161	157	157
query20	117	113	111	111
query21	216	110	107	107
query22	4857	4257	4690	4257
query23	34980	34056	34009	34009
query24	5969	2892	2908	2892
query25	511	410	397	397
query26	626	157	157	157
query27	1582	284	284	284
query28	4322	2434	2410	2410
query29	672	431	417	417
query30	234	156	158	156
query31	1000	787	812	787
query32	70	53	55	53
query33	450	304	303	303
query34	912	486	491	486
query35	881	763	704	704
query36	1068	898	917	898
query37	140	85	80	80
query38	4034	3849	3967	3849
query39	1478	1425	1436	1425
query40	209	96	94	94
query41	50	47	50	47
query42	119	100	94	94
query43	502	473	484	473
query44	1160	818	794	794
query45	193	165	165	165
query46	1135	754	753	753
query47	1931	1807	1873	1807
query48	449	362	374	362
query49	692	414	411	411
query50	845	401	408	401
query51	7066	7019	7020	7019
query52	98	88	88	88
query53	249	181	185	181
query54	568	478	452	452
query55	75	73	73	73
query56	263	237	271	237
query57	1256	1106	1106	1106
query58	220	224	250	224
query59	3032	2960	3127	2960
query60	286	264	254	254
query61	108	102	104	102
query62	755	666	650	650
query63	221	177	179	177
query64	1318	646	607	607
query65	3250	3213	3201	3201
query66	690	301	308	301
query67	16101	15705	15651	15651
query68	4146	585	594	585
query69	713	422	316	316
query70	1164	1102	1109	1102
query71	517	274	271	271
query72	7724	4008	4098	4008
query73	805	325	328	325
query74	9724	8964	8972	8964
query75	3989	2639	2663	2639
query76	3619	866	869	866
query77	749	307	296	296
query78	9927	9388	9229	9229
query79	1188	539	538	538
query80	729	431	457	431
query81	532	244	238	238
query82	423	136	143	136
query83	331	133	140	133
query84	294	81	80	80
query85	944	280	283	280
query86	398	297	295	295
query87	4381	4328	4374	4328
query88	3264	2333	2342	2333
query89	394	279	285	279
query90	2255	191	186	186
query91	183	141	143	141
query92	61	46	47	46
query93	2254	529	514	514
query94	822	292	298	292
query95	344	252	246	246
query96	616	283	276	276
query97	3236	3154	3110	3110
query98	219	205	200	200
query99	1516	1330	1281	1281
Total cold run time: 314236 ms
Total hot run time: 195622 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0333f8aa163b66b718fbfae983644b3e44e2c9d, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.02	0.02
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.51	0.51
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.03	0.03	0.02
query9	0.56	0.51	0.49
query10	0.55	0.56	0.55
query11	0.14	0.12	0.12
query12	0.13	0.10	0.10
query13	0.60	0.60	0.59
query14	2.97	3.03	2.97
query15	0.89	0.82	0.81
query16	0.40	0.38	0.39
query17	1.04	1.08	1.04
query18	0.19	0.19	0.19
query19	1.96	1.89	2.03
query20	0.01	0.00	0.00
query21	15.37	0.58	0.57
query22	2.57	2.07	1.87
query23	17.51	0.84	0.81
query24	3.40	1.05	1.39
query25	0.31	0.07	0.16
query26	0.46	0.13	0.13
query27	0.03	0.03	0.05
query28	10.12	1.11	1.07
query29	12.58	3.23	3.16
query30	0.25	0.06	0.06
query31	2.87	0.38	0.39
query32	3.28	0.46	0.46
query33	2.99	3.06	3.01
query34	16.65	4.43	4.44
query35	4.47	4.46	4.40
query36	0.67	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.11 s
Total hot run time: 32.9 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.25% (9596/25758)
Line Coverage: 28.64% (79288/276806)
Region Coverage: 28.11% (41046/146005)
Branch Coverage: 24.74% (20911/84528)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f0333f8aa163b66b718fbfae983644b3e44e2c9d_f0333f8aa163b66b718fbfae983644b3e44e2c9d/report/index.html

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants