Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sampleimprove2.0 #40982

Draft
wants to merge 3 commits into
base: branch-2.0
Choose a base branch
from
Draft

Sampleimprove2.0 #40982

wants to merge 3 commits into from

Conversation

Jibing-Li
Copy link
Contributor

backport: #40811

@Jibing-Li
[improvement](statistics)Return -1 to neredis if report olap table ro…
c4c94c8 
…w count for new table is not done for all tablets. (apache#40457)

Return -1 to neredis if report olap table row count for new table is not
done for all tablets.
After this change, nereids could know new table is empty or not. When
it's not empty but not reported yet, return -1 as row count to nereids.
@Jibing-Li
[fix](statistics)Fix drop stats log editlog bug. Catch drop stats exc…
79a282a 
…eption while truncate table. (apache#40738)

Bug fix.
1. Fix drop stats log editlog NPE bug.
2. Catch drop stats exception while truncate table.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.84% (8141/21513)
Line Coverage: 29.59% (66998/226413)
Region Coverage: 29.08% (34566/118880)
Branch Coverage: 24.98% (17809/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4744e2730117160c2618416c9bc7fbc4777a4563_4744e2730117160c2618416c9bc7fbc4777a4563/report/index.html

@Jibing-Li
[fix](statistics)Fix empty table keep auto analyze bug. (apache#40811)
0efd0e6 
1. Fix empty table keep auto analyze bug, reset new partition flag no
matter row count is 0 or not.
2. Use report row count == -1 to check table is empty or not.
@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49018 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0efd0e67507cb340e7a71d67d1b5afaa8e9c4483, data reload: false

------ Round 1 ----------------------------------
q1	18034	4361	4356	4356
q2	2037	150	143	143
q3	10341	1932	1971	1932
q4	10109	1262	1359	1262
q5	8730	3920	3931	3920
q6	231	123	125	123
q7	2040	1603	1571	1571
q8	9493	2734	2707	2707
q9	13213	9938	9790	9790
q10	8606	3529	3505	3505
q11	421	239	239	239
q12	464	306	297	297
q13	18311	3947	4053	3947
q14	356	316	320	316
q15	503	456	455	455
q16	555	441	455	441
q17	1137	985	961	961
q18	7339	6882	6784	6784
q19	1695	1618	1541	1541
q20	509	310	311	310
q21	4423	4099	4043	4043
q22	505	398	375	375
Total cold run time: 119052 ms
Total hot run time: 49018 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4292	4296	4280	4280
q2	320	230	217	217
q3	4190	4125	4138	4125
q4	2759	2757	2731	2731
q5	7076	7070	7090	7070
q6	236	116	118	116
q7	3269	2798	2811	2798
q8	4348	4376	4437	4376
q9	13623	13618	13522	13522
q10	4224	4230	4291	4230
q11	731	715	680	680
q12	1027	844	831	831
q13	7218	3742	3726	3726
q14	451	427	421	421
q15	497	462	456	456
q16	625	591	596	591
q17	3792	3829	3783	3783
q18	8692	8675	8721	8675
q19	1713	1663	1666	1663
q20	2389	2103	2110	2103
q21	8441	8566	8332	8332
q22	1030	939	925	925
Total cold run time: 80943 ms
Total hot run time: 75651 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.84% (8141/21513)
Line Coverage: 29.59% (67003/226413)
Region Coverage: 29.08% (34568/118880)
Branch Coverage: 24.99% (17815/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0efd0e67507cb340e7a71d67d1b5afaa8e9c4483_0efd0e67507cb340e7a71d67d1b5afaa8e9c4483/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212176 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0efd0e67507cb340e7a71d67d1b5afaa8e9c4483, data reload: false

query1	915	389	409	389
query2	6550	2198	2114	2114
query3	6918	214	207	207
query4	23226	21661	21528	21528
query5	19741	6503	6584	6503
query6	280	224	222	222
query7	4332	304	301	301
query8	258	264	262	262
query9	3055	2647	2583	2583
query10	467	312	312	312
query11	16632	14881	15403	14881
query12	134	77	79	77
query13	1051	432	443	432
query14	17141	13436	13778	13436
query15	367	220	250	220
query16	6466	286	257	257
query17	1807	939	942	939
query18	885	313	309	309
query19	213	150	149	149
query20	85	73	78	73
query21	196	99	101	99
query22	5210	5142	4968	4968
query23	34216	33444	33405	33405
query24	7840	6357	6334	6334
query25	515	432	439	432
query26	1263	163	161	161
query27	2411	293	291	291
query28	6118	2269	2230	2230
query29	2856	2670	2734	2670
query30	253	172	170	170
query31	959	717	760	717
query32	74	58	66	58
query33	462	267	269	267
query34	881	470	467	467
query35	1148	897	916	897
query36	1399	1192	1230	1192
query37	178	60	60	60
query38	3063	2959	2953	2953
query39	1384	1302	1317	1302
query40	306	93	92	92
query41	40	46	37	37
query42	87	96	92	92
query43	632	569	557	557
query44	1171	736	722	722
query45	245	232	234	232
query46	1231	948	943	943
query47	1936	1886	1850	1850
query48	503	405	417	405
query49	649	380	395	380
query50	881	633	576	576
query51	4735	4650	4652	4650
query52	94	90	89	89
query53	234	188	180	180
query54	2674	2465	2488	2465
query55	92	84	90	84
query56	234	209	199	199
query57	1182	1099	1183	1099
query58	220	211	219	211
query59	3531	3253	3169	3169
query60	212	209	197	197
query61	96	93	95	93
query62	786	464	466	464
query63	208	175	182	175
query64	3501	1611	1513	1513
query65	3628	3556	3564	3556
query66	765	407	428	407
query67	15618	14870	16495	14870
query68	9894	631	637	631
query69	518	285	262	262
query70	1646	1454	1325	1325
query71	406	306	308	306
query72	6869	4861	4841	4841
query73	768	328	329	328
query74	6237	5794	5909	5794
query75	5289	3721	3735	3721
query76	5661	1141	1195	1141
query77	963	256	255	255
query78	12658	11759	12042	11759
query79	6948	662	636	636
query80	1559	403	382	382
query81	492	236	240	236
query82	1427	100	98	98
query83	177	135	133	133
query84	261	68	68	68
query85	878	321	317	317
query86	330	293	292	292
query87	3245	2996	2991	2991
query88	4523	2380	2384	2380
query89	371	290	293	290
query90	1959	220	226	220
query91	155	125	130	125
query92	57	53	52	52
query93	5587	610	546	546
query94	702	207	208	207
query95	2023	1931	2032	1931
query96	645	342	337	337
query97	6457	6423	6329	6329
query98	236	225	211	211
query99	2976	935	858	858
Total cold run time: 318357 ms
Total hot run time: 212176 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0efd0e67507cb340e7a71d67d1b5afaa8e9c4483, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.79	0.06	0.06
query5	0.53	0.53	0.51
query6	1.23	0.64	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.61	0.61
query14	0.79	0.79	0.79
query15	0.78	0.75	0.77
query16	0.36	0.39	0.37
query17	1.00	1.00	1.01
query18	0.22	0.24	0.25
query19	1.93	1.83	1.83
query20	0.01	0.01	0.01
query21	15.47	0.58	0.56
query22	2.31	2.24	1.73
query23	17.14	0.96	1.08
query24	7.61	1.18	0.92
query25	0.39	0.08	0.05
query26	0.89	0.16	0.16
query27	0.04	0.03	0.03
query28	5.48	0.79	0.74
query29	12.77	2.38	2.21
query30	0.56	0.54	0.51
query31	2.82	0.39	0.38
query32	3.35	0.49	0.50
query33	3.08	3.06	3.06
query34	15.23	4.79	4.79
query35	4.88	4.84	4.85
query36	1.08	1.02	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.15
query41	0.06	0.02	0.01
query42	0.03	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.43 s
Total hot run time: 30.8 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0efd0e67507cb340e7a71d67d1b5afaa8e9c4483 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Jibing-Li @doris-robot