Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) #40973

Merged
merged 1 commit into from
Sep 26, 2024
Merged

[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) #40973

merged 1 commit into from
Sep 26, 2024

Conversation

Jibing-Li
Copy link
Contributor

backport: #40457

@Jibing-Li
[improvement](statistics)Return -1 to neredis if report olap table ro…
c4c94c8 
…w count for new table is not done for all tablets. (apache#40457)

Return -1 to neredis if report olap table row count for new table is not
done for all tablets.
After this change, nereids could know new table is empty or not. When
it's not empty but not reported yet, return -1 as row count to nereids.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Sep 19, 2024
@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 48936 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4c94c8006dedaa0a92f01aece99b3d055d90763, data reload: false

------ Round 1 ----------------------------------
q1	17816	4333	4320	4320
q2	2068	152	141	141
q3	10274	1913	1860	1860
q4	10344	1279	1276	1276
q5	8315	3804	3927	3804
q6	229	124	123	123
q7	2026	1611	1603	1603
q8	9248	2715	2698	2698
q9	10200	9786	9877	9786
q10	8654	3500	3505	3500
q11	419	235	246	235
q12	467	298	304	298
q13	18398	3971	4030	3971
q14	346	336	329	329
q15	504	448	459	448
q16	549	454	455	454
q17	1153	971	951	951
q18	7406	6846	6871	6846
q19	1678	1529	1580	1529
q20	551	320	293	293
q21	4435	4181	4101	4101
q22	485	370	396	370
Total cold run time: 115565 ms
Total hot run time: 48936 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4298	4297	4297
q2	321	225	226	225
q3	4153	4140	4115	4115
q4	2766	2738	2741	2738
q5	7182	7045	7114	7045
q6	234	117	118	117
q7	3208	2842	2869	2842
q8	4339	4432	4443	4432
q9	13603	13457	13601	13457
q10	4252	4291	4218	4218
q11	716	688	699	688
q12	1012	850	870	850
q13	6853	3723	3682	3682
q14	457	418	419	418
q15	496	456	464	456
q16	614	593	615	593
q17	3855	3815	3772	3772
q18	8726	8618	8659	8618
q19	1721	1666	1681	1666
q20	2382	2116	2125	2116
q21	8360	8359	8381	8359
q22	1025	903	881	881
Total cold run time: 80588 ms
Total hot run time: 75585 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.84% (8141/21513)
Line Coverage: 29.59% (66996/226413)
Region Coverage: 29.08% (34567/118880)
Branch Coverage: 24.99% (17812/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c4c94c8006dedaa0a92f01aece99b3d055d90763_c4c94c8006dedaa0a92f01aece99b3d055d90763/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 214841 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4c94c8006dedaa0a92f01aece99b3d055d90763, data reload: false

query1	931	415	437	415
query2	6552	2287	2270	2270
query3	6918	216	208	208
query4	23856	21761	21920	21761
query5	19745	6800	6586	6586
query6	278	226	228	226
query7	4335	319	313	313
query8	241	260	246	246
query9	3092	2624	2567	2567
query10	464	304	305	304
query11	15768	15148	14764	14764
query12	127	74	73	73
query13	1035	432	434	432
query14	17189	13135	13787	13135
query15	384	234	221	221
query16	6479	287	268	268
query17	1726	924	895	895
query18	888	327	331	327
query19	210	151	149	149
query20	81	79	76	76
query21	190	103	97	97
query22	5250	5029	5091	5029
query23	34650	33778	33838	33778
query24	7791	6331	6336	6331
query25	537	427	424	424
query26	1269	162	170	162
query27	2382	302	296	296
query28	6111	2251	2215	2215
query29	2781	2777	2683	2683
query30	248	174	173	173
query31	967	739	755	739
query32	74	61	61	61
query33	468	269	271	269
query34	897	476	485	476
query35	1123	894	922	894
query36	1454	1144	1316	1144
query37	178	61	60	60
query38	3030	2903	2911	2903
query39	1396	1355	1338	1338
query40	309	99	97	97
query41	39	39	38	38
query42	83	95	90	90
query43	614	585	585	585
query44	1156	723	715	715
query45	248	229	227	227
query46	1241	969	994	969
query47	1883	1831	1736	1736
query48	524	408	405	405
query49	667	388	379	379
query50	850	611	611	611
query51	4855	4656	4719	4656
query52	89	90	87	87
query53	236	190	180	180
query54	2694	2506	2535	2506
query55	95	84	88	84
query56	232	229	220	220
query57	1244	1120	1162	1120
query58	222	206	217	206
query59	3456	3277	3495	3277
query60	216	216	213	213
query61	103	96	98	96
query62	829	471	493	471
query63	212	180	185	180
query64	3529	1600	1379	1379
query65	3709	3650	3576	3576
query66	773	403	448	403
query67	15803	16278	16901	16278
query68	9596	673	664	664
query69	498	279	268	268
query70	1720	1383	1372	1372
query71	410	316	322	316
query72	6965	4618	4958	4618
query73	783	317	324	317
query74	6346	5868	5836	5836
query75	5316	3730	3723	3723
query76	5441	1139	1202	1139
query77	919	265	252	252
query78	13008	14404	12367	12367
query79	7459	664	654	654
query80	768	393	403	393
query81	509	252	256	252
query82	268	96	101	96
query83	171	139	142	139
query84	255	71	71	71
query85	731	330	325	325
query86	340	301	328	301
query87	3283	3077	3072	3072
query88	3822	2374	2386	2374
query89	392	281	296	281
query90	1932	212	215	212
query91	157	134	127	127
query92	66	53	53	53
query93	2017	553	564	553
query94	709	215	214	214
query95	2018	1996	1927	1927
query96	665	338	335	335
query97	6515	6501	6414	6414
query98	238	230	216	216
query99	2879	875	868	868
Total cold run time: 313202 ms
Total hot run time: 214841 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4c94c8006dedaa0a92f01aece99b3d055d90763, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.04
query4	1.81	0.08	0.06
query5	0.53	0.52	0.52
query6	1.28	0.61	0.63
query7	0.01	0.01	0.02
query8	0.04	0.03	0.02
query9	0.54	0.49	0.47
query10	0.54	0.53	0.55
query11	0.12	0.09	0.09
query12	0.12	0.10	0.10
query13	0.62	0.61	0.61
query14	0.80	0.79	0.78
query15	0.79	0.77	0.77
query16	0.39	0.37	0.35
query17	1.02	1.02	1.01
query18	0.20	0.26	0.23
query19	1.93	1.87	1.84
query20	0.01	0.01	0.02
query21	15.49	0.58	0.58
query22	2.34	2.30	1.67
query23	17.59	1.05	0.94
query24	5.06	1.70	1.02
query25	0.37	0.12	0.04
query26	0.62	0.17	0.16
query27	0.05	0.04	0.03
query28	7.33	0.72	0.73
query29	12.64	2.30	2.35
query30	0.56	0.54	0.53
query31	2.80	0.38	0.37
query32	3.39	0.49	0.48
query33	3.06	3.10	3.05
query34	15.25	4.79	4.78
query35	4.84	4.85	4.82
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.03	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.91 s
Total hot run time: 30.84 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c4c94c8006dedaa0a92f01aece99b3d055d90763 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@Jibing-Li Jibing-Li marked this pull request as ready for review September 26, 2024 02:30
@Jibing-Li Jibing-Li merged commit 2f5be08 into apache:branch-2.0 Sep 26, 2024
21 of 24 checks passed
@Jibing-Li Jibing-Li deleted the row2.0 branch September 26, 2024 02:32
Jibing-Li added a commit that referenced this pull request Oct 8, 2024
@Jibing-Li
[improvement](statistics)Return -1 to neredis if report olap table ro…
769f2c0 
…w count for new table is not done for all tablets. (#40457) (#40973)

backport: #40457
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Jibing-Li @doris-robot