Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](json-quote) fix json quote func for not find the func #42005

Merged
merged 2 commits into from
Oct 17, 2024

Conversation

amorynan
Copy link
Contributor

@amorynan amorynan commented Oct 17, 2024

pick #39931

…9931)

1.  fix function not found
before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844e ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Oct 17, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.84% (8155/21553)
Line Coverage: 29.56% (67106/227030)
Region Coverage: 29.04% (34601/119167)
Branch Coverage: 24.99% (17859/71478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c17f9f478303bafc835f54637c093b3fc5d6dae0_c17f9f478303bafc835f54637c093b3fc5d6dae0/report/index.html

@amorynan amorynan closed this Oct 17, 2024
@amorynan amorynan reopened this Oct 17, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@amorynan
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49048 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37f88a0426550e3d76a41bc5e1bcddcf8108d2af, data reload: false

------ Round 1 ----------------------------------
q1	17662	4379	4371	4371
q2	2060	152	143	143
q3	10320	1959	1894	1894
q4	10108	1250	1332	1250
q5	8414	3879	3833	3833
q6	228	123	123	123
q7	2028	1641	1604	1604
q8	9356	2725	2696	2696
q9	10386	9869	9881	9869
q10	8675	3510	3486	3486
q11	424	238	241	238
q12	474	288	293	288
q13	18336	3967	4050	3967
q14	350	319	327	319
q15	512	455	454	454
q16	566	458	454	454
q17	1120	956	941	941
q18	7250	6933	6845	6845
q19	1689	1607	1503	1503
q20	538	312	290	290
q21	4374	4084	4081	4081
q22	511	406	399	399
Total cold run time: 115381 ms
Total hot run time: 49048 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4262	4325	4262
q2	321	225	235	225
q3	4151	4124	4138	4124
q4	2739	2719	2743	2719
q5	7128	7111	7057	7057
q6	234	119	114	114
q7	3179	2812	2837	2812
q8	4324	4437	4460	4437
q9	13662	13521	13606	13521
q10	4209	4259	4258	4258
q11	730	677	672	672
q12	1019	853	836	836
q13	7031	3725	3736	3725
q14	457	433	423	423
q15	501	456	452	452
q16	611	593	563	563
q17	3733	3818	3936	3818
q18	8829	8826	8773	8773
q19	1701	1610	1618	1610
q20	2378	2123	2111	2111
q21	8316	8391	8360	8360
q22	1014	977	953	953
Total cold run time: 80596 ms
Total hot run time: 75825 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.86% (8159/21553)
Line Coverage: 29.58% (67146/227030)
Region Coverage: 29.06% (34627/119167)
Branch Coverage: 25.01% (17875/71478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/37f88a0426550e3d76a41bc5e1bcddcf8108d2af_37f88a0426550e3d76a41bc5e1bcddcf8108d2af/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212955 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37f88a0426550e3d76a41bc5e1bcddcf8108d2af, data reload: false

query1	921	390	443	390
query2	6572	2174	2197	2174
query3	6920	200	207	200
query4	23211	21593	21716	21593
query5	19720	6516	6526	6516
query6	280	226	253	226
query7	4168	296	307	296
query8	247	244	248	244
query9	3045	2647	2561	2561
query10	405	317	301	301
query11	16408	15048	15034	15034
query12	127	76	74	74
query13	1013	426	435	426
query14	17289	13717	13426	13426
query15	389	221	227	221
query16	6473	273	260	260
query17	1776	912	921	912
query18	893	311	313	311
query19	214	148	148	148
query20	103	98	96	96
query21	188	98	94	94
query22	5338	5100	5015	5015
query23	34314	33400	33554	33400
query24	7818	6270	6331	6270
query25	513	429	416	416
query26	1257	162	167	162
query27	2494	293	287	287
query28	6048	2254	2218	2218
query29	2927	2621	2702	2621
query30	251	170	165	165
query31	971	723	775	723
query32	75	64	60	60
query33	446	258	253	253
query34	866	490	475	475
query35	1156	918	915	915
query36	1169	1310	1087	1087
query37	174	57	58	57
query38	3048	2933	2914	2914
query39	1392	1308	1316	1308
query40	310	93	96	93
query41	39	38	37	37
query42	93	93	83	83
query43	609	611	538	538
query44	1155	715	714	714
query45	239	227	228	227
query46	1226	956	947	947
query47	1789	1692	1849	1692
query48	482	405	407	405
query49	648	375	372	372
query50	858	642	633	633
query51	4805	4640	4597	4597
query52	83	88	87	87
query53	231	192	186	186
query54	2645	2463	2492	2463
query55	90	86	87	86
query56	229	211	203	203
query57	1317	1095	1151	1095
query58	232	214	210	210
query59	3520	3160	3418	3160
query60	218	214	194	194
query61	99	93	97	93
query62	856	470	490	470
query63	196	177	168	168
query64	3595	1596	1454	1454
query65	3619	3543	3588	3543
query66	769	406	406	406
query67	15824	18023	16233	16233
query68	9210	647	637	637
query69	485	279	268	268
query70	1648	1357	1370	1357
query71	396	289	308	289
query72	6800	4772	4797	4772
query73	757	331	349	331
query74	6236	5815	5761	5761
query75	5131	3774	3708	3708
query76	5216	1135	1152	1135
query77	881	264	244	244
query78	12451	11718	11846	11718
query79	8950	640	642	640
query80	1615	409	400	400
query81	487	240	237	237
query82	1669	102	97	97
query83	172	135	133	133
query84	259	70	69	69
query85	893	314	310	310
query86	335	290	281	281
query87	3232	3082	3018	3018
query88	4707	2315	2311	2311
query89	479	326	269	269
query90	1897	218	218	218
query91	163	134	125	125
query92	59	49	51	49
query93	6295	566	549	549
query94	651	211	210	210
query95	1934	1958	1971	1958
query96	651	319	324	319
query97	6472	6303	6320	6303
query98	227	216	196	196
query99	2958	811	805	805
Total cold run time: 319711 ms
Total hot run time: 212955 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37f88a0426550e3d76a41bc5e1bcddcf8108d2af, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.05
query4	1.81	0.07	0.06
query5	0.54	0.52	0.52
query6	1.28	0.60	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.50	0.47
query10	0.54	0.52	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.61	0.60	0.62
query14	0.78	0.79	0.79
query15	0.78	0.78	0.77
query16	0.37	0.38	0.36
query17	1.02	1.03	1.01
query18	0.24	0.25	0.24
query19	1.96	1.83	1.89
query20	0.02	0.01	0.02
query21	15.43	0.56	0.56
query22	1.81	2.11	1.78
query23	17.02	1.11	1.05
query24	3.81	1.07	1.58
query25	0.36	0.10	0.06
query26	0.51	0.15	0.15
query27	0.04	0.04	0.04
query28	8.61	0.71	0.70
query29	12.78	2.28	2.31
query30	0.60	0.50	0.53
query31	2.82	0.38	0.38
query32	3.39	0.51	0.50
query33	3.11	3.06	3.06
query34	15.25	4.80	4.81
query35	4.85	4.86	4.87
query36	1.08	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.93 s
Total hot run time: 31.12 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 37f88a0426550e3d76a41bc5e1bcddcf8108d2af with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@eldenmoon eldenmoon changed the title [fix](json-quote) fix json quote func for not find the func (#39931) [fix](json-quote) fix json quote func for not find the func Oct 17, 2024
@eldenmoon eldenmoon merged commit 59d3326 into apache:branch-2.0 Oct 17, 2024
20 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants