Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump (#40940) #40976

Merged

Conversation

liaoxin01
Copy link
Contributor

cherry pick from #40940

…pache#40940)

The VARCHAR(0) created has an actual length of 0, and BE didn't restrict
it, leading to a BE core dump.
to have a length of 65533. This PR restricts data import when the length
is 0 to avoid a core dump.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49109 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2ce6f3a2c1aa877e398eef542775c8eab7f82c3, data reload: false

------ Round 1 ----------------------------------
q1	18201	4391	4363	4363
q2	2035	150	145	145
q3	10271	1913	1978	1913
q4	10101	1258	1326	1258
q5	8386	3843	3910	3843
q6	233	123	124	123
q7	2082	1577	1615	1577
q8	9561	2705	2705	2705
q9	12427	9831	9764	9764
q10	8646	3568	3523	3523
q11	411	253	261	253
q12	465	296	303	296
q13	18389	3957	4044	3957
q14	362	333	327	327
q15	500	458	455	455
q16	540	455	474	455
q17	1125	902	947	902
q18	7253	7013	6945	6945
q19	1667	1578	1513	1513
q20	520	325	318	318
q21	4421	4081	4086	4081
q22	489	399	393	393
Total cold run time: 118085 ms
Total hot run time: 49109 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4309	4321	4312	4312
q2	321	227	220	220
q3	4139	4177	4117	4117
q4	2755	2732	2741	2732
q5	7111	7104	7120	7104
q6	242	117	119	117
q7	3258	2814	2850	2814
q8	4370	4445	4457	4445
q9	13689	13647	13621	13621
q10	4233	4267	4247	4247
q11	744	679	656	656
q12	1029	865	849	849
q13	6727	3754	3675	3675
q14	452	430	432	430
q15	500	465	454	454
q16	623	583	593	583
q17	3903	3836	3834	3834
q18	8742	8918	8733	8733
q19	1756	1685	1692	1685
q20	2327	2137	2110	2110
q21	8515	8472	8393	8393
q22	1021	918	926	918
Total cold run time: 80766 ms
Total hot run time: 76049 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.85% (8143/21513)
Line Coverage: 29.60% (67020/226411)
Region Coverage: 29.09% (34582/118880)
Branch Coverage: 25.00% (17820/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a2ce6f3a2c1aa877e398eef542775c8eab7f82c3_a2ce6f3a2c1aa877e398eef542775c8eab7f82c3/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212992 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2ce6f3a2c1aa877e398eef542775c8eab7f82c3, data reload: false

query1	944	398	426	398
query2	6528	2245	2053	2053
query3	6922	212	206	206
query4	23702	21749	21629	21629
query5	19737	6558	6526	6526
query6	292	228	219	219
query7	4148	309	314	309
query8	236	256	261	256
query9	3084	2647	2588	2588
query10	413	303	276	276
query11	15753	15355	14992	14992
query12	127	75	75	75
query13	1028	438	442	438
query14	17039	13873	13098	13098
query15	363	210	226	210
query16	5828	286	259	259
query17	1782	935	902	902
query18	896	330	325	325
query19	215	164	154	154
query20	81	75	79	75
query21	195	99	99	99
query22	5313	5073	5080	5073
query23	34103	33641	33590	33590
query24	7855	6337	6365	6337
query25	540	419	416	416
query26	1349	163	163	163
query27	2513	299	290	290
query28	6128	2237	2215	2215
query29	2998	2622	2637	2622
query30	245	179	179	179
query31	933	713	759	713
query32	70	61	61	61
query33	458	264	262	262
query34	873	479	475	475
query35	1127	943	913	913
query36	1157	1166	1344	1166
query37	168	59	60	59
query38	3090	2900	2933	2900
query39	1382	1335	1331	1331
query40	311	100	96	96
query41	39	37	37	37
query42	86	83	84	83
query43	642	570	548	548
query44	1184	723	725	723
query45	242	229	229	229
query46	1235	982	962	962
query47	2134	1713	1674	1674
query48	501	412	416	412
query49	647	370	376	370
query50	845	585	620	585
query51	4779	4629	4693	4629
query52	88	83	86	83
query53	234	184	181	181
query54	2697	2473	2503	2473
query55	87	79	85	79
query56	222	206	207	206
query57	1137	1097	1086	1086
query58	220	205	212	205
query59	3861	3356	3324	3324
query60	232	199	196	196
query61	101	98	99	98
query62	834	513	487	487
query63	203	179	176	176
query64	3584	1587	1491	1491
query65	3638	3582	3564	3564
query66	795	419	451	419
query67	16156	15848	15652	15652
query68	6825	642	643	642
query69	483	286	265	265
query70	1625	1392	1527	1392
query71	376	312	309	309
query72	6748	4844	4813	4813
query73	757	321	327	321
query74	6317	5815	5849	5815
query75	4658	3792	3744	3744
query76	4055	1157	1218	1157
query77	551	257	272	257
query78	12378	11828	11592	11592
query79	5448	653	661	653
query80	2601	402	388	388
query81	520	245	242	242
query82	1569	104	97	97
query83	246	137	137	137
query84	259	70	71	70
query85	1406	318	330	318
query86	349	285	299	285
query87	3212	3038	3017	3017
query88	4626	2384	2373	2373
query89	360	296	289	289
query90	1735	218	210	210
query91	164	131	136	131
query92	55	53	51	51
query93	3102	581	558	558
query94	891	211	213	211
query95	2008	2032	1948	1948
query96	647	346	331	331
query97	6393	6460	6394	6394
query98	227	213	209	209
query99	2785	873	853	853
Total cold run time: 309725 ms
Total hot run time: 212992 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2ce6f3a2c1aa877e398eef542775c8eab7f82c3, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.05	0.04
query4	1.82	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.61	0.60
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.53	0.49	0.48
query10	0.54	0.54	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.63	0.61	0.60
query14	0.77	0.78	0.79
query15	0.79	0.76	0.74
query16	0.37	0.39	0.38
query17	1.03	1.04	0.98
query18	0.21	0.28	0.23
query19	1.87	1.85	1.88
query20	0.01	0.01	0.01
query21	15.47	0.55	0.55
query22	2.03	2.29	1.38
query23	16.76	1.09	0.92
query24	7.95	0.82	0.89
query25	0.38	0.11	0.05
query26	0.78	0.14	0.15
query27	0.03	0.05	0.03
query28	5.53	0.73	0.74
query29	12.63	2.36	2.40
query30	0.58	0.55	0.52
query31	2.82	0.40	0.37
query32	3.34	0.50	0.49
query33	3.09	3.05	3.12
query34	15.25	4.81	4.78
query35	4.84	4.84	4.86
query36	1.09	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.88 s
Total hot run time: 30.37 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a2ce6f3a2c1aa877e398eef542775c8eab7f82c3 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@dataroaring dataroaring merged commit a6be436 into apache:branch-2.0 Sep 19, 2024
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants