Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](table size) Fix MoW table merge data fault #40880

Merged
merged 1 commit into from
Sep 19, 2024
Merged

[Fix](table size) Fix MoW table merge data fault #40880

merged 1 commit into from
Sep 19, 2024

Conversation

Yukang-Lian
Copy link
Collaborator

Proposed changes

Issue #40879 step 1

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42439 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c1964f0bbbbb9e8108862e6d6546c6cb81ed6232, data reload: false

------ Round 1 ----------------------------------
q1	18403	7595	7333	7333
q2	2678	165	233	165
q3	11702	1257	1298	1257
q4	10276	889	886	886
q5	8119	3175	3237	3175
q6	243	156	153	153
q7	1047	626	601	601
q8	9430	1934	2044	1934
q9	6851	6421	6422	6421
q10	7067	2391	2361	2361
q11	442	250	254	250
q12	403	224	217	217
q13	17775	3030	2973	2973
q14	243	212	213	212
q15	584	532	517	517
q16	498	430	414	414
q17	988	924	923	923
q18	7293	6736	6848	6736
q19	1397	1243	1217	1217
q20	585	288	282	282
q21	3984	3563	3417	3417
q22	1099	995	1014	995
Total cold run time: 111107 ms
Total hot run time: 42439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7162	7146	7162	7146
q2	335	225	241	225
q3	2888	2842	2845	2842
q4	1956	1858	1880	1858
q5	5390	5406	5398	5398
q6	233	138	144	138
q7	2065	1699	1688	1688
q8	3184	3298	3337	3298
q9	8350	8404	8381	8381
q10	3417	3406	3420	3406
q11	566	481	478	478
q12	757	613	571	571
q13	6839	3011	2993	2993
q14	314	285	271	271
q15	565	521	518	518
q16	508	461	462	461
q17	1735	1678	1671	1671
q18	7994	7735	7693	7693
q19	1705	1684	1664	1664
q20	2053	1792	1784	1784
q21	5247	5086	5063	5063
q22	1090	1006	1008	1006
Total cold run time: 64353 ms
Total hot run time: 58553 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9576/25662)
Line Coverage: 28.69% (79094/275707)
Region Coverage: 28.18% (40971/145371)
Branch Coverage: 24.80% (20880/84186)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c1964f0bbbbb9e8108862e6d6546c6cb81ed6232_c1964f0bbbbb9e8108862e6d6546c6cb81ed6232/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 193766 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c1964f0bbbbb9e8108862e6d6546c6cb81ed6232, data reload: false

query1	938	358	382	358
query2	6535	1756	1782	1756
query3	6703	208	220	208
query4	29560	23820	24006	23820
query5	4731	515	504	504
query6	275	179	174	174
query7	4632	311	312	311
query8	281	220	224	220
query9	8600	2560	2562	2560
query10	523	291	296	291
query11	16829	15362	15505	15362
query12	157	97	101	97
query13	1673	414	403	403
query14	11654	6646	6761	6646
query15	218	168	168	168
query16	7619	474	456	456
query17	1574	546	535	535
query18	2014	288	290	288
query19	204	137	138	137
query20	116	113	107	107
query21	215	99	103	99
query22	4510	4108	4188	4108
query23	34509	34047	33803	33803
query24	9461	3115	3071	3071
query25	672	400	391	391
query26	1171	161	155	155
query27	2202	285	288	285
query28	6436	2080	2070	2070
query29	913	403	404	403
query30	307	147	144	144
query31	991	724	810	724
query32	103	52	60	52
query33	760	297	283	283
query34	928	463	476	463
query35	856	779	717	717
query36	1036	903	918	903
query37	154	78	73	73
query38	4093	3918	3906	3906
query39	1470	1436	1381	1381
query40	234	92	93	92
query41	50	46	47	46
query42	118	97	95	95
query43	491	465	444	444
query44	1198	781	765	765
query45	195	166	169	166
query46	1114	827	813	813
query47	1869	1778	1772	1772
query48	437	349	352	349
query49	1147	398	395	395
query50	926	426	430	426
query51	7182	7046	6858	6858
query52	101	90	86	86
query53	257	183	177	177
query54	802	457	451	451
query55	77	76	78	76
query56	269	256	264	256
query57	1207	1077	1109	1077
query58	246	227	238	227
query59	2710	2627	2521	2521
query60	292	268	266	266
query61	101	103	102	102
query62	919	677	666	666
query63	212	184	180	180
query64	4603	616	613	613
query65	3238	3141	3168	3141
query66	1251	292	298	292
query67	16111	15635	15601	15601
query68	3189	877	858	858
query69	463	346	366	346
query70	1155	1150	1134	1134
query71	352	335	344	335
query72	6081	3552	3725	3552
query73	595	594	596	594
query74	9181	9005	9174	9005
query75	3140	2971	2946	2946
query76	1974	851	853	851
query77	422	361	353	353
query78	9370	9198	9318	9198
query79	919	904	868	868
query80	629	612	600	600
query81	445	244	243	243
query82	200	194	194	194
query83	169	166	171	166
query84	256	112	103	103
query85	734	427	414	414
query86	333	312	313	312
query87	4409	4363	4464	4363
query88	4231	4160	4283	4160
query89	370	357	371	357
query90	1432	310	310	310
query91	159	160	164	160
query92	78	74	72	72
query93	1041	1046	1028	1028
query94	596	384	350	350
query95	450	413	415	413
query96	485	482	478	478
query97	3120	3107	3124	3107
query98	235	224	215	215
query99	1528	1308	1296	1296
Total cold run time: 280721 ms
Total hot run time: 193766 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c1964f0bbbbb9e8108862e6d6546c6cb81ed6232, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.02	0.02
query3	0.23	0.07	0.06
query4	1.64	0.10	0.09
query5	0.51	0.50	0.52
query6	1.12	0.72	0.74
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.51	0.51
query10	0.56	0.58	0.57
query11	0.15	0.11	0.10
query12	0.15	0.10	0.10
query13	0.60	0.59	0.60
query14	1.45	1.42	1.51
query15	0.87	0.85	0.83
query16	0.38	0.37	0.39
query17	1.02	1.05	0.99
query18	0.21	0.20	0.20
query19	1.88	1.78	1.80
query20	0.01	0.00	0.00
query21	15.39	0.54	0.53
query22	3.18	4.45	3.12
query23	17.89	1.01	1.06
query24	2.33	0.23	0.22
query25	0.23	0.13	0.06
query26	0.16	0.13	0.13
query27	0.03	0.04	0.03
query28	12.43	1.01	1.00
query29	12.53	3.36	3.29
query30	0.24	0.07	0.07
query31	2.87	0.37	0.39
query32	3.27	0.46	0.46
query33	3.00	2.98	3.01
query34	15.43	4.26	4.30
query35	4.33	4.33	4.33
query36	0.70	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.95 s
Total hot run time: 31.56 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 07a5362 into apache:master Sep 19, 2024
24 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Sep 19, 2024
@Yukang-Lian @dataroaring
[Fix](table size) Fix MoW table merge data fault (#40880)
02ee7ae 
Issue #40879 step 1
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
@Yukang-Lian Yukang-Lian mentioned this pull request Nov 11, 2024
Merged
16 tasks
gavinchou pushed a commit that referenced this pull request Nov 13, 2024
@Yukang-Lian
[cherry-pick](branch-2.1) Pick "[Fix](table size) Fix MoW table merge…
6ecd55f 
… data fault (#40880)" (#43610)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Yukang-Lian @doris-robot @gavinchou @dataroaring @wm1581066