Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](scan) fix predicate contains cast that results in null, the predicate will be miss. (#39550) #40503

Merged
merged 1 commit into from
Sep 7, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Sep 7, 2024

…edicate will be miss. (#39550)
#39550

drop table datetest;

create table datetest (
  id int,
  dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
  "replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
+------------+
| dt         |
+------------+
| 2024-01-01 |
+------------+

now

mysql [test10]>select dt from datetest  WHERE dt = 1 ;
Empty set (0.16 sec)

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Sep 7, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Mryange
Copy link
Contributor Author

Mryange commented Sep 7, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49178 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cb96320f96f1133f717edba75cbf1a1301181b96, data reload: false

------ Round 1 ----------------------------------
q1	17620	4409	4348	4348
q2	2073	155	152	152
q3	10260	1890	1930	1890
q4	10187	1231	1319	1231
q5	8394	3916	3935	3916
q6	226	123	126	123
q7	2040	1646	1608	1608
q8	9565	2738	2709	2709
q9	13368	9785	9773	9773
q10	8666	3552	3511	3511
q11	403	247	246	246
q12	471	298	302	298
q13	18343	3947	4020	3947
q14	361	342	324	324
q15	502	465	449	449
q16	518	448	461	448
q17	1124	972	955	955
q18	7300	6930	6949	6930
q19	1668	1506	1511	1506
q20	543	326	295	295
q21	4468	4146	4124	4124
q22	493	400	395	395
Total cold run time: 118593 ms
Total hot run time: 49178 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4329	4271	4271
q2	318	222	231	222
q3	4188	4168	4177	4168
q4	2769	2749	2754	2749
q5	7195	7172	7086	7086
q6	241	120	120	120
q7	3261	2805	2834	2805
q8	4374	4441	4479	4441
q9	13738	13659	13625	13625
q10	4245	4264	4234	4234
q11	754	682	680	680
q12	1049	849	855	849
q13	7239	3764	3792	3764
q14	451	434	420	420
q15	508	468	459	459
q16	629	593	602	593
q17	3855	3852	3817	3817
q18	8890	8796	8728	8728
q19	1730	1665	1699	1665
q20	2364	2126	2137	2126
q21	8493	8414	8564	8414
q22	1029	975	968	968
Total cold run time: 81627 ms
Total hot run time: 76204 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8133/21498)
Line Coverage: 29.58% (66948/226340)
Region Coverage: 29.07% (34542/118840)
Branch Coverage: 24.98% (17803/71268)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cb96320f96f1133f717edba75cbf1a1301181b96_cb96320f96f1133f717edba75cbf1a1301181b96/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 213044 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cb96320f96f1133f717edba75cbf1a1301181b96, data reload: false

query1	934	398	411	398
query2	6552	2355	2111	2111
query3	6922	204	199	199
query4	23515	21698	21827	21698
query5	19726	6517	6538	6517
query6	293	217	229	217
query7	4338	305	305	305
query8	259	267	243	243
query9	3068	2649	2594	2594
query10	421	290	304	290
query11	15786	14982	15309	14982
query12	128	78	73	73
query13	1016	439	439	439
query14	17374	13006	13686	13006
query15	366	225	229	225
query16	6472	281	263	263
query17	1762	953	917	917
query18	894	317	312	312
query19	207	151	151	151
query20	82	80	79	79
query21	192	96	91	91
query22	5292	5012	4816	4816
query23	34290	33478	33316	33316
query24	6875	6280	6267	6267
query25	524	438	441	438
query26	1006	168	170	168
query27	2306	304	297	297
query28	6068	2292	2241	2241
query29	2841	2675	2845	2675
query30	246	164	168	164
query31	956	742	772	742
query32	74	59	59	59
query33	437	274	257	257
query34	857	473	474	473
query35	1171	939	908	908
query36	1529	1314	1230	1230
query37	92	57	60	57
query38	3108	2939	2911	2911
query39	1411	1327	1325	1325
query40	226	98	91	91
query41	41	37	37	37
query42	98	86	91	86
query43	630	658	564	564
query44	1164	719	724	719
query45	245	235	225	225
query46	1234	976	962	962
query47	1890	1776	1800	1776
query48	493	409	405	405
query49	623	382	400	382
query50	859	613	620	613
query51	4751	4733	4681	4681
query52	97	84	76	76
query53	234	192	186	186
query54	2646	2513	2454	2454
query55	88	88	85	85
query56	212	204	213	204
query57	1239	1164	1172	1164
query58	218	215	220	215
query59	3371	3434	3397	3397
query60	213	207	201	201
query61	95	91	92	91
query62	755	463	489	463
query63	202	179	178	178
query64	5036	1590	1345	1345
query65	3616	3551	3581	3551
query66	642	420	420	420
query67	15878	15146	17176	15146
query68	9294	647	658	647
query69	519	272	264	264
query70	1584	1349	1419	1349
query71	405	312	309	309
query72	6951	4790	4746	4746
query73	750	315	322	315
query74	6343	5930	5848	5848
query75	4894	3696	3707	3696
query76	4896	1009	1039	1009
query77	681	250	248	248
query78	12543	12814	13003	12814
query79	11974	663	639	639
query80	754	387	374	374
query81	505	242	237	237
query82	1086	100	98	98
query83	175	134	128	128
query84	258	70	71	70
query85	807	311	309	309
query86	336	296	321	296
query87	3287	3065	3019	3019
query88	5284	2348	2329	2329
query89	429	289	305	289
query90	2561	204	212	204
query91	175	132	131	131
query92	58	53	51	51
query93	6107	589	582	582
query94	1308	212	206	206
query95	2048	1974	2008	1974
query96	657	343	328	328
query97	6479	6388	6289	6289
query98	224	210	199	199
query99	2854	884	876	876
Total cold run time: 322412 ms
Total hot run time: 213044 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cb96320f96f1133f717edba75cbf1a1301181b96, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.79	0.08	0.09
query5	0.53	0.52	0.53
query6	1.23	0.61	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.53	0.50	0.46
query10	0.52	0.55	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.80	0.78	0.80
query15	0.78	0.78	0.76
query16	0.38	0.36	0.36
query17	1.03	1.02	1.02
query18	0.22	0.26	0.24
query19	1.92	1.87	1.79
query20	0.02	0.01	0.01
query21	15.43	0.55	0.55
query22	1.99	2.77	1.64
query23	17.12	1.04	0.87
query24	4.68	0.96	1.48
query25	0.40	0.08	0.06
query26	0.54	0.17	0.16
query27	0.04	0.05	0.04
query28	8.11	0.75	0.74
query29	12.65	2.30	2.31
query30	0.61	0.55	0.52
query31	2.80	0.38	0.37
query32	3.37	0.50	0.49
query33	3.09	3.12	3.11
query34	15.27	4.78	4.80
query35	4.86	4.83	4.84
query36	1.09	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.46 s
Total hot run time: 30.77 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cb96320f96f1133f717edba75cbf1a1301181b96 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@Mryange
Copy link
Contributor Author

Mryange commented Sep 7, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49004 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aebb2bd28a15c3aa47f0345c71ed51961f13e5de, data reload: false

------ Round 1 ----------------------------------
q1	17908	4345	4310	4310
q2	2061	149	142	142
q3	10434	1906	1920	1906
q4	10334	1249	1322	1249
q5	8705	3921	4054	3921
q6	233	124	124	124
q7	2030	1646	1596	1596
q8	9274	2706	2697	2697
q9	9945	9821	9823	9821
q10	8627	3526	3482	3482
q11	413	251	250	250
q12	465	302	303	302
q13	18330	3990	4013	3990
q14	349	329	326	326
q15	516	450	467	450
q16	534	457	448	448
q17	1131	961	891	891
q18	7289	6780	6871	6780
q19	1701	1551	1551	1551
q20	522	325	337	325
q21	4408	4159	4061	4061
q22	501	382	404	382
Total cold run time: 115710 ms
Total hot run time: 49004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4296	4285	4368	4285
q2	318	218	226	218
q3	4145	4115	4110	4110
q4	2754	2752	2752	2752
q5	7200	7086	7095	7086
q6	237	119	120	119
q7	3264	2865	2862	2862
q8	4336	4404	4459	4404
q9	13602	13512	13545	13512
q10	4241	4252	4286	4252
q11	755	745	654	654
q12	1009	850	829	829
q13	6786	3701	3769	3701
q14	453	420	420	420
q15	498	464	450	450
q16	630	585	590	585
q17	3790	3819	3826	3819
q18	8691	8759	8798	8759
q19	1723	1677	1638	1638
q20	2373	2113	2128	2113
q21	8570	8408	8311	8311
q22	1010	908	918	908
Total cold run time: 80681 ms
Total hot run time: 75787 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8133/21498)
Line Coverage: 29.58% (66952/226335)
Region Coverage: 29.07% (34542/118837)
Branch Coverage: 24.98% (17803/71268)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aebb2bd28a15c3aa47f0345c71ed51961f13e5de_aebb2bd28a15c3aa47f0345c71ed51961f13e5de/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212214 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aebb2bd28a15c3aa47f0345c71ed51961f13e5de, data reload: false

query1	909	397	408	397
query2	6549	2224	2016	2016
query3	6917	206	204	204
query4	23601	21771	21678	21678
query5	21208	6547	6666	6547
query6	286	222	226	222
query7	4280	302	325	302
query8	265	262	239	239
query9	3088	2654	2595	2595
query10	449	308	292	292
query11	15700	15127	15834	15127
query12	132	82	83	82
query13	1049	461	451	451
query14	19730	13334	13646	13334
query15	375	222	227	222
query16	7680	283	280	280
query17	2932	982	909	909
query18	2386	315	318	315
query19	211	153	154	153
query20	78	74	75	74
query21	193	94	99	94
query22	5203	5060	4984	4984
query23	34172	33388	33569	33388
query24	6767	6299	6363	6299
query25	522	429	423	423
query26	765	163	163	163
query27	2259	301	291	291
query28	6151	2280	2240	2240
query29	2906	2800	2831	2800
query30	248	168	166	166
query31	932	743	728	728
query32	74	61	61	61
query33	459	262	253	253
query34	867	475	473	473
query35	1097	924	876	876
query36	1365	1281	1218	1218
query37	91	60	60	60
query38	3048	2863	2989	2863
query39	1370	1328	1319	1319
query40	207	100	97	97
query41	40	37	37	37
query42	81	85	87	85
query43	730	685	636	636
query44	1183	726	728	726
query45	239	233	229	229
query46	1220	956	965	956
query47	1817	1733	1794	1733
query48	489	414	419	414
query49	613	382	380	380
query50	849	605	594	594
query51	4742	4695	4711	4695
query52	86	82	85	82
query53	244	194	179	179
query54	2659	2473	2461	2461
query55	87	84	83	83
query56	226	211	216	211
query57	1165	1098	1185	1098
query58	221	218	220	218
query59	3647	3300	3444	3300
query60	207	214	209	209
query61	100	95	95	95
query62	828	499	463	463
query63	223	178	178	178
query64	2860	1538	1442	1442
query65	3602	3549	3558	3549
query66	739	426	407	407
query67	15993	15191	17740	15191
query68	8506	662	682	662
query69	499	258	253	253
query70	1637	1495	1257	1257
query71	423	302	309	302
query72	6917	4757	4589	4589
query73	757	325	318	318
query74	6329	5834	5892	5834
query75	4562	3695	3651	3651
query76	4532	1169	1178	1169
query77	543	255	250	250
query78	12490	12104	11504	11504
query79	10099	649	663	649
query80	2913	390	388	388
query81	539	240	234	234
query82	1522	96	95	95
query83	174	132	133	132
query84	269	69	72	69
query85	1467	331	314	314
query86	358	294	310	294
query87	3196	2998	2966	2966
query88	5101	2331	2299	2299
query89	479	288	281	281
query90	1882	225	216	216
query91	169	125	131	125
query92	60	48	51	48
query93	6978	587	598	587
query94	1034	205	204	204
query95	2022	1996	2054	1996
query96	644	324	325	324
query97	6531	6286	6333	6286
query98	218	209	194	194
query99	2932	841	838	838
Total cold run time: 327193 ms
Total hot run time: 212214 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aebb2bd28a15c3aa47f0345c71ed51961f13e5de, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.04
query4	1.81	0.07	0.07
query5	0.53	0.53	0.51
query6	1.26	0.62	0.61
query7	0.01	0.01	0.00
query8	0.04	0.03	0.02
query9	0.52	0.51	0.48
query10	0.54	0.53	0.54
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.78	0.79	0.78
query15	0.76	0.77	0.78
query16	0.38	0.37	0.39
query17	1.01	1.01	0.99
query18	0.23	0.24	0.26
query19	1.93	1.74	1.84
query20	0.02	0.01	0.01
query21	15.45	0.56	0.56
query22	2.15	2.00	1.75
query23	17.41	1.05	1.01
query24	7.51	1.69	0.95
query25	0.38	0.13	0.06
query26	0.73	0.16	0.16
query27	0.04	0.03	0.04
query28	5.09	0.72	0.70
query29	12.76	2.26	2.28
query30	0.58	0.54	0.52
query31	2.81	0.38	0.37
query32	3.39	0.49	0.49
query33	3.09	3.04	3.09
query34	15.24	4.80	4.78
query35	4.85	4.88	4.83
query36	1.09	1.02	1.01
query37	0.06	0.05	0.04
query38	0.03	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 103.97 s
Total hot run time: 30.74 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit aebb2bd28a15c3aa47f0345c71ed51961f13e5de with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@xiaokang xiaokang merged commit d154c68 into apache:branch-2.0 Sep 7, 2024
19 of 23 checks passed
@xiaokang xiaokang changed the title [fix](scan) fix predicate contains cast that results in null, the pr… [fix](scan) fix predicate contains cast that results in null, the predicate will be miss. (#39550) Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants