[improve](task) Support splitting agent batch tasks automatically #42703

[w41ter]

cherry pick from #42703

[w41ter]

run buildall

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[doris-robot]

TPC-H: Total hot run time: 49189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca4317750f11e9c944129afba56b35c5ac6c30f5, data reload: false

------ Round 1 ----------------------------------
q1	17850	4367	4305	4305
q2	2061	155	142	142
q3	10352	1910	1943	1910
q4	10367	1257	1310	1257
q5	8559	3978	3861	3861
q6	225	123	126	123
q7	2048	1610	1616	1610
q8	9292	2729	2695	2695
q9	10578	10016	9963	9963
q10	8622	3520	3510	3510
q11	430	252	238	238
q12	467	295	295	295
q13	18338	3980	4043	3980
q14	354	327	332	327
q15	500	457	462	457
q16	526	458	474	458
q17	1115	988	922	922
q18	7294	6781	7033	6781
q19	1700	1578	1561	1561
q20	547	332	314	314
q21	4442	4122	4094	4094
q22	495	386	398	386
Total cold run time: 116162 ms
Total hot run time: 49189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4344	4339	4339
q2	319	232	224	224
q3	4125	4141	4123	4123
q4	2724	2715	2742	2715
q5	7182	7172	7078	7078
q6	246	125	120	120
q7	3202	2806	2834	2806
q8	4391	4473	4520	4473
q9	14199	14059	14057	14057
q10	4205	4270	4253	4253
q11	753	711	708	708
q12	1011	854	849	849
q13	6525	3752	3726	3726
q14	448	428	423	423
q15	498	466	450	450
q16	646	590	589	589
q17	3805	3812	3814	3812
q18	8729	8830	8858	8830
q19	1710	1663	1641	1641
q20	2380	2141	2096	2096
q21	8337	8429	8398	8398
q22	999	928	928	928
Total cold run time: 80747 ms
Total hot run time: 76638 ms

[doris-robot]

TPC-DS: Total hot run time: 212750 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca4317750f11e9c944129afba56b35c5ac6c30f5, data reload: false

query1	939	394	406	394
query2	6556	2237	2162	2162
query3	6944	203	225	203
query4	23005	21758	21672	21672
query5	19720	6528	6509	6509
query6	292	223	243	223
query7	4332	308	307	307
query8	274	258	242	242
query9	3120	2664	2579	2579
query10	468	308	306	306
query11	15509	15184	15276	15184
query12	128	77	75	75
query13	1029	450	429	429
query14	17324	13149	13528	13149
query15	365	226	227	226
query16	6503	280	259	259
query17	1784	954	873	873
query18	898	325	318	318
query19	224	156	163	156
query20	105	93	98	93
query21	197	95	101	95
query22	5253	5027	5016	5016
query23	34276	33401	33614	33401
query24	7652	6314	6351	6314
query25	540	438	442	438
query26	1278	165	165	165
query27	2291	311	298	298
query28	6090	2242	2211	2211
query29	2958	2680	2794	2680
query30	244	166	163	163
query31	952	739	762	739
query32	73	59	60	59
query33	452	262	276	262
query34	877	481	482	481
query35	1151	950	929	929
query36	1288	1229	1080	1080
query37	90	60	63	60
query38	3079	2909	2888	2888
query39	1372	1340	1331	1331
query40	308	99	95	95
query41	41	38	38	38
query42	86	95	94	94
query43	641	529	597	529
query44	1186	719	725	719
query45	247	234	236	234
query46	1218	965	970	965
query47	1756	1920	1716	1716
query48	496	414	407	407
query49	649	386	395	386
query50	854	591	612	591
query51	4786	4736	4694	4694
query52	99	91	80	80
query53	224	178	190	178
query54	2649	2441	2460	2441
query55	96	79	99	79
query56	226	220	213	213
query57	1332	1198	1208	1198
query58	224	226	208	208
query59	3506	3301	3359	3301
query60	219	222	215	215
query61	98	95	99	95
query62	844	498	427	427
query63	198	184	177	177
query64	3522	1598	1512	1512
query65	3660	3550	3548	3548
query66	779	418	421	418
query67	17560	15294	17687	15294
query68	9403	651	654	651
query69	491	267	278	267
query70	1610	1544	1285	1285
query71	413	308	317	308
query72	6819	4887	4893	4887
query73	759	315	321	315
query74	6221	5882	5794	5794
query75	5089	3721	3732	3721
query76	5057	1150	1168	1150
query77	726	257	263	257
query78	12845	11724	12572	11724
query79	13636	638	640	638
query80	1305	388	390	388
query81	497	233	236	233
query82	1439	97	98	97
query83	172	134	131	131
query84	258	72	70	70
query85	896	329	320	320
query86	340	285	296	285
query87	3275	3013	3062	3013
query88	5178	2287	2281	2281
query89	481	299	314	299
query90	2000	225	213	213
query91	172	126	124	124
query92	61	52	53	52
query93	6458	601	559	559
query94	791	216	210	210
query95	2028	2002	2008	2002
query96	638	334	317	317
query97	6506	6425	6349	6349
query98	229	205	214	205
query99	2862	866	784	784
Total cold run time: 325791 ms
Total hot run time: 212750 ms

[doris-robot]

ClickBench: Total hot run time: 30.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca4317750f11e9c944129afba56b35c5ac6c30f5, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.04
query4	1.78	0.07	0.09
query5	0.56	0.53	0.51
query6	1.30	0.60	0.60
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.50	0.50	0.48
query10	0.54	0.54	0.53
query11	0.11	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.80	0.80
query15	0.77	0.75	0.76
query16	0.37	0.37	0.36
query17	1.01	1.00	1.02
query18	0.21	0.25	0.24
query19	1.94	1.76	1.86
query20	0.02	0.01	0.01
query21	15.48	0.55	0.54
query22	2.19	2.61	1.31
query23	16.86	1.07	1.15
query24	4.27	5.81	0.55
query25	0.37	0.09	0.05
query26	0.91	0.17	0.15
query27	0.04	0.03	0.03
query28	4.65	0.73	0.73
query29	12.65	2.30	2.25
query30	0.58	0.52	0.52
query31	2.82	0.39	0.37
query32	3.37	0.50	0.49
query33	3.07	3.04	3.04
query34	15.24	4.81	4.80
query35	4.88	4.86	4.88
query36	1.05	1.02	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.13	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 99.81 s
Total hot run time: 30.02 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ca4317750f11e9c944129afba56b35c5ac6c30f5 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s