Release Notes 3.0.4

[gavinchou]

Behavior Changes

• In the Audit log, the force flag is retained for drop table and drop database statements. ([fix](drop sql) add force in the tosql for drop table and drop database #43227)
• When exporting data to Parquet/ORC formats, the bitmap, quantile_state, and hll types are exported in Binary format. Additionally, support has been added for exporting jsonb and variant types, which are exported as string. ([fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #44041)
  • Related documentation: [Export Overview - Apache Doris](https://doris.apache.org/docs/3.0/data-operate/export/export-overview)
• The Hudi JNI Scanner has been replaced from Spark API to Hadoop API to enhance compatibility. Users can switch by setting the session variable set hudi_jni_scanner=spark/hadoop. ([fix](hudi) upgrade hudi to 0.15.0 #44267)
• The use of auto bucket in Colocate tables is prohibited. ([fix](table) Disable create, alter auto bucket table with colocate #44396)
• Paimon cache has been added to the Catalog, eliminating real-time data queries. ([feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911)
• The default value of max_broker_concurrency has been increased to improve performance for large-scale data imports with Broker Load. ([performance](load) increase max_broker_concurrency to 100 #44929)
• The default value of the storage medium for Auto Partition partitions has been changed to the attribute value of the current table's storage medium, rather than using the system default value. ([Bug](auto-partition) fix auto partition could set storage_medium properties #45955)
• Column updates are prohibited during Schema Change execution for Key columns. ([Fix](partial update) abort partial update on shadow index's tablet when the including columns miss key columns on new schema #46347)
• For Key columns containing auto-increment columns, support has been added to allow column updates without providing the auto-increment column. ([opt](auto-inc) Allow to miss auto-increment column and other value columns in partial update #44528)
• The FE ID generator strategy has been switched to a time-based approach, and IDs no longer start from 10000. ([chore](conf) Set enable_advance_next_id=true by default #44790)
• In the compute-storage separation mode, the default stale rowset recycling delay for Compaction has been reduced to 1800 seconds to decrease the recycling interval. This may cause large queries to fail in extreme scenarios, and adjustments can be made as needed. ([Enhancement](config) Modify cloud default stale rowset recycle time #45460)
• The show cache hotspot statement has been disabled in compute-storage separation mode, and direct access to system tables is required. ([chore](file cache) Disable show cache hotspot stmt #47332)
• Deleting the system-created admin user is prohibited. ([fix](auth)Prohibit deleting admin user #44751)

Improvements

Storage

• Optimized the issue of Routine Load tasks frequently timing out due to a small max_match_interval setting. ([improve](routine load) introduce routine load task min timeout #46292)
• Improved performance for Broker Load when importing multiple compressed files. (branch-3.0: [performance](load) fix broker load scan ranges for unsplittable files #43161 #43975)
• Increased the default value of webserver_num_workers to enhance Stream Load performance. (branch-3.0: [opt](load) Increase the default value of webserver_num_workers #46304 #46593)
• Optimized the load imbalance issue for Routine Load import tasks during BE node scaling. (branch-3.0: [improve](routine load) ensure load balance after scaling up BE nodes #44693 #44798)
• Improved the use of Routine Load thread pools to prevent timeouts from affecting queries. (branch-3.0: [fix](routine load) replace heavy work pool with routine load thread pool for metadata fetching #44907 #45039)

Compute-Storage Separation

• Enhanced the stability and observability of the Meta-service. ([improve][ms] Bvars add the FDB get_count_normalized indicator #44036, [opt](recycler) Improve robustness and observability  #45617, [Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers #45255, [opt](cloud) reset ms response message in case it is reused for retry #45068)
• Optimized File Cache by adding an early eviction strategy, reducing lock time, and improving query performance. ([enhancement](cloud) file cache evict in advance #47473, [enhancement](cloud) add profile counter for file cache #45678, [fix](cloud) shorten cache lock held time and add metrics #47472)
• Improved initialization checks and queue transitions for File Cache to enhance stability. ([fix](cloud) explicit message when parse file_cache_path error #44004, [fix](cloud) serialize cache init to avoid unstable cache pick #44429, [fix](cloud) fix CHECK failed when transmit from non-TTL to normal #45057, [chore](file_cache) Set enbale_dump_error_file to false by default #47229)
• Increased the speed of HDFS data recycling. ([Fix](recycler) Fix recycler Hdfs accessor list all bug #46393)
• Optimized performance issues when the FE acquires compute groups during ultra-high-frequency imports. ([opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() #47203)
• Improved several import-related parameters for primary key tables in compute-storage separation to enhance the stability of real-time high-concurrency imports. ([fix](cloud-mow) Make delete bitmap cache expired time more reasonable #47295, [improve](cloud-mow)Make some mow calculating delete bitmap config more reasonable #46750, [fix](cloud-mow) Make some timeout about mow more reasonable #46365)

Lakehouse

• Supported reading Hive tables in JSON format. ([feature](hive)support hive catalog read json table.  #43469)
  • Related documentation: [Text/CSV/JSON - Apache Doris](https://doris.apache.org/docs/dev/lakehouse/file-formats/text#json)
• Introduced the session variable enable_text_validate_utf8 to skip UTF-8 encoding checks for CSV formats. ([enchement](utf8)import enable_text_validate_utf8 session var #45537)
  • Related documentation: [Text/CSV/JSON - Apache Doris](https://doris.apache.org/docs/dev/lakehouse/file-formats/text#character-set)
• Updated the Hudi version to 0.15 and optimized query planning performance for Hudi tables.
• Improved read performance for MaxCompute partitioned tables. ([enchement](mc)Optimize reading of maxcompute partition tables. #45148)
• Optimized performance for Parquet file delayed materialization under high filter rates. (branch-2.1: [fix](parquet-reader) Fixed the issue of excessive scanning data in late materialization‌ case of parquet reader #46121 #46183)
• Supported delayed materialization for complex Parquet types. ([opt](parquet-reader)Implement late materialization of parquet complex types. #44098)
• Optimized predicate pushdown logic for ORC types, supporting more predicate conditions for index filtering. ([enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255)

Asynchronous Materialized Views

• Supported more scenarios for aggregate roll-up rewriting. ([opt](mtmv) Support any_value aggregate function rollup rewrite when exsits aggregate materialized view #44412)

Query Optimizer

• Improved partition pruning performance. ([opt](nereids) improve prune partition with lots of in (xxx) #46261)
• Added rules to eliminate group by keys based on data characteristics. ([feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform #43391)
• Adaptively adjusted the wait time for Runtime Filters based on the target table size. ([feat](nereids)set runtime filter wait time according to table row count and table type #42640)
• Improved the ability to push down aggregations in joins to fit more scenarios. ([opt](nereids) enhance PUSH_DOWN_AGG_THROUGH_JOIN_ONE_SIDE #43856, [opt](nereids) support pushing down aggr distinct through join #43380)
• Improved Limit pushdown for aggregations to fit more scenarios. ([opt](nereids) optimize push limit to agg #44042)

Others

• Optimized startup scripts for FE, BE, and MS processes to provide clearer output. ([chore](script) fix start_fe.sh --version not work and MetaService scripts occur error in Debian GNU/Linux 11 (bullseye) #45610, [chore](version) Show binary version in metrics: fe be ms #45490, [chore](bash) optimize output information when doris_cloud startup  #45883)
• The case sensitivity of table names in show tables now matches MySQL behavior. ([fix](show)show tables should be case insensitive when lowerCaseTableNames is 1 or 2. #46030)
• show index now supports arbitrary target table types. ([opt](show) let all types table support show index #45861)
• information_schema.columns now supports displaying default values. ([improvement](information_schema)Support show default value in information_schema. #44849)
• information_schema.views now supports displaying view definitions. ([improvement](information_schema)Show view definition in information_schema.views. #45857)
• Supported the MySQL protocol COM_RESET_CONNECTION command. ([improvement](mysql)Support mysql COM_RESET_CONNECTION command. #44747)

Bug Fixes

Storage

• Fixed potential memory errors during the import process for aggregate table models. ([bugfix](memtable) arena is freed early and will cause use after free #46997)
• Resolved the issue of Routine Load offset loss during FE master node restarts in compute-storage separation mode. (branch-3.0: [fix](cloud) fix routine load loss data when fe master node restart #46149 #46566)
• Fixed memory leaks in FE Observer nodes during batch import scenarios in compute-storage mode. (branch-3.0: [fix](cloud)(bulk load) fix memory leak in FE observer node #47074 #47244)
• Resolved the issue of Cumulative Point rollback during Full Compaction with Order Data Compaction. ([Fix](full compaction) Full compaction should not do ordered data compaction #44359)
• Fixed the issue where Delete operations could temporarily prevent Tablet Compaction scheduling. ([Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466)
• Resolved incorrect Tablet states after Schema Change in multi-compute-cluster scenarios. ([enhancement](meta) Sync tablet meta even if local state is not running #45821)
• Fixed the potential NPE error when performing Column Rename Schema Change on primary key tables with sequence_type. ([Fix](schema change) Fix NPE when rename column on table which has sequence type column #46906)
• Data Correctness: Fixed correctness issues for primary key tables when importing partial column updates containing DELETE SIGN columns. ([Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194)
• Resolved potential memory leaks in FE when Publish tasks for primary key tables were continuously stuck. ([Fix](TPartitionVersionInfo) Fix duplicate TPartitionVersionInfo in PublishVersionTask.partitionVersionInfos #44846)

Compute-Storage Separation

• Fixed the issue where File Cache size could exceed the table data size. ([fix](cloud) fix file cache potential leakage #46561, [fix](cloud) Support clean tablet file cache when tablet drop #46390)

• Resolved upload failures at the 5MB boundary during data uploads. ([fix](s3filewriter) Fix s3_write_buffer_size boundary issue #47333)

• Enhanced robustness by adding more parameter checks for several alter operations in Storage Vault. ([fix](storage vault) Fix missing use_path_style when create storage vault #45155, [fix](vault) avoid encrypt twice when altering vault #45156, [fix](vault) fix CreateTableLikeStmt cannot work in stoarge vault mode #46625, [opt](storage vault) Check s3.root.path cannot be empty #47078, [fix](vault) Fix bugs about altering storage vault name #45685, [fix](vault) Fix creating storage vault failed with azure backend #46779)

• Resolved issues with data recycling failures or slow recycling due to improper Storage Vault configurations. ([fix](recycler) Fix premature exit recycling when there is an invalid storage vault #46798, [Fix](recycler) Fix retain inverted indexes in tmp rowset recycling #47536, [Fix](recycler) Delete again to double check when recycle tablet failed by some bugs #47475, [Fix](recycler) Fix recycler fail when dealing with rowset [0-1] #47324, [fix](recycler) Process self-defined domain names for s3 storage vault #45072)

• Fixed the issue where data recycling could stall, preventing timely recycling. ([fix](recycler) Fix CountdownEvent error and hang #45760)

• Resolved incorrect retries for MTTM-230 errors in compute-storage separation mode. ([fix](cloud) Fix async mtmv job retry when meet -230 in cloud #47370, [fix](cloud) Fix cloud -230 retry not reset ctx state #47326)

• Fixed the issue where Group Commit WAL was not fully replayed during BE decommissioning in compute-storage separation mode. ([fix](cloud) Fix cloud decomission and check wal #47187)

• Resolved the issue where Tablet Meta exceeding 2GB rendered MS unavailable. ([fix](meta-service) Avoid rowset meta exceeds 2G result in protobuf fatal #44780)

• Data Correctness: Fixed two duplicate Key issues in primary key tables in compute-storage separation mode. ([Fix](merge-on-write) Should update pending delete bitmap KVs in MS when no need to calc delete bitmaps in publish phase  #46039, [Fix](merge-on-write) Should clear GetDeleteBitmapUpdateLockResponse when geting delete bitmap update lock fail and retry #44975)

• Resolved the issue where Base Compaction could continuously fail due to large Delete Bitmaps in primary key tables during high-frequency real-time imports. ([fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap #46969)

• Modified incorrect retry logic for Schema Change in primary key tables in compute-storage separation mode to enhance robustness. ([fix](cloud-mow) schema change should retry when encouter TXN_CONFILCT in cloud mode #46748)

Lakehouse

Hive

• Fixed the issue where Hive views created by Spark could not be queried. (branch-2.1: [fix](hive) support query hive view created by spark #43553)
  • Resolved the issue where certain Hive Transaction tables could not be read correctly. ([fix](hive)fix hive insert only translaction table. #45753)
  • Fixed the issue where partition pruning failed for Hive tables with special characters in partitions. ([fix](hive)fix hive catalog miss partition that have special characters. #42906)

Iceberg

• Fixed the issue where Iceberg tables could not be created in Kerberos authentication environments. ([feat](catalog)Support Pre-Execution Authentication for HMS Type Iceberg Catalog Operations.  #43445)
• Resolved the issue where count(*) queries were inaccurate for Iceberg tables with dangling deletes. ([fix](iceberg)Fix count(*) error with dangling delete problem #44039)
• Fixed the issue where query errors occurred due to mismatched column names in Iceberg tables. ([fix](iceberg)Bring field_id with parquet files And fix map type's key optional #44470)
• Resolved the issue where Iceberg tables could not be read after partition modifications. ([enchement](iceberg)support read iceberg partition evolution table. #45367)

Paimon

• Fixed the issue where Paimon Catalog could not access Alibaba Cloud OSS-HDFS. ([Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS #42585)

Hudi

• Fixed the issue where partition pruning failed for Hudi tables in certain scenarios. ([fix](hudi)Add hudi catalog read partition table partition prune #44669)

JDBC

• Fixed the issue where tables could not be retrieved using JDBC Catalog after enabling case-insensitive table names.

MaxCompute

• Fixed the issue where partition pruning failed for MaxCompute tables in certain scenarios. ([enchement](maxcompute)add mc catalog read partition table partition prune #44508)

Others

• Fixed the issue where Export tasks caused memory leaks in FE. ([fix](Export) fix a memory leak in the FE because of the ExportJob #44019)
• Resolved the issue where S3 object storage could not be accessed via HTTPS protocol. ([fix](s3) do not replace https scheme if specified #44242)
• Fixed the issue where Kerberos authentication tickets could not be automatically refreshed. ([feat](catalog)Replace HadoopUGI with HadoopKerberosAuthenticator to Support Kerberos Ticket Auto-Renewal #44916)
• Resolved the issue where reading Hadoop Block compressed format files failed. ([fix](hive) fix block decompressor bug #45289)
• When querying ORC format data, CHAR type predicates are no longer pushed down to avoid potential result errors. ([Fix](ORC) Not push down fixed char type in orc reader  #45484)

Asynchronous Materialized Views

• Fixed the issue where transparent query rewriting could lead to planning or result errors in extreme scenarios. ([fix](mtmv) Fix filter position different but same causing rewritten by materialized view fail #44575, [fix](mtmv) Fix mv is deleted in nested mv causing query err and fix some test #45744)
• Resolved the issue where multiple build tasks could be generated during asynchronous materialized view scheduling in extreme scenarios. ([fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020, [Improve](mtmv) skip the generation of invalid task for refresh mtmv #46280)

Query Optimizer

• Fixed the issue where some expression rewrites could produce incorrect expressions. ([fix](nereids) fix UnknownValue's reference in simplify range rule #44637 #44770, branch-3.0: [fix](nereids) fix compare with long min for simplify comparison rule #44920, branch-3.0: [fix](nereids) fix compare with date like overflow #45868 #45922, branch-3.0: [fix](nereids) fix compare with date like literal #45382 #45596)
• Resolved occasional incorrect results from SQL Cache. ([fix](cache) fix same sql return wrong result when switch database with use db and enable sql cache #44782, [fix](sql_cache) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') #44631, [fix](nereids) fix sql cache bug and some tests #46443, branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47266)
• Fixed the issue where Limit pushdown for aggregation operators could produce incorrect results in some scenarios. ([fix](Nereids) set correct sort key for aggregate  #45369)
• Resolved the issue where delayed materialization optimization could produce incorrect execution plans in some scenarios. ([fix](nereids) support one phase DeferMaterializeTopN #45693, [fix](nereids) topN filter: use ObjectId as map key instead of Topn node  #46551)

Query Execution

• Fixed the issue where regular expressions and like functions produced incorrect results with special characters. ([fix](hyperscan) Fix hyper scan fall back to re2 #44547)
• Resolved the issue where SQL Cache results could be incorrect when switching databases. ([fix](cache) fix same sql return wrong result when switch database with use db and enable sql cache #44782)
• Fixed a series of Arrow Flight-related issues. ([fix](arrow-flight-sql) Fix query result is empty and not return query error message #45023, [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929)
• Resolved the issue where results were incorrect when the Hash table for HashJoin exceeded 4GB in some cases. ([Bug](join) fix columnstr64's offset overflow on serialize_value_into_arena #46461)
• Fixed the overflow issue of the convert_to function with Chinese characters. ([fix](mem) heap-buffer-overflow for function convert_to #46405)
• Resolved the issue where results could be incorrect in extreme scenarios when group by was used with Limit. ([Bug](fix) Fix topn agg limit may get error result in when refresh heap #47844)
• Fixed the issue where results could be incorrect when accessing certain system tables. ([fix](local shuffle) Set serial execution for schema scan operator #47498)
• Resolved the issue where the percentile function could cause system crashes. ([Fix](bug) Percentile* func core when percent args is negative number #47068)
• Fixed the performance degradation issue for single-table queries with Limit. (branch-3.0: [fix](scan) Fix scan with limit #46035 #46090)
• Resolved the issue where StDistanceSphere and StAngleSphere functions caused system crashes. ([Fix](function) fix coredump of function StDistanceSphere and StAngleSphere #45508)
• Fixed the issue where map_agg results were incorrect. ([Bug](map) fix wrong result on map_agg with streaming agg #40454)

Semi-structured Data Management

BloomFilter Index

• Fixed the exception caused by large parameters in BloomFilter Index. ([fix](bf index) add ngram bf index validation in nereids index definition check #45780)
• Resolved the issue of high memory usage during BloomFilter Index writes. ([opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833)
• Fixed the issue where BloomFilter Index was not correctly deleted when columns were dropped. ([fix](bloom filter)Fix drop column with bloom filter index #44361, [fix](bloom filter)Fix drop column with bloom filter (#41369) #43378)

Inverted Index

• Fixed the occasional crash during inverted index construction. ([fix] (build index) fix build index coredump #43246)
• Resolved the issue where words with zero occurrences occupied space during inverted index merging. ([fix](index compaction)Skip writing terms with a doc frequency of 0 #43113)
• Prevented abnormal large values in Index Size statistics. ([fix](index size) discard index size when meta size is invalid #46549)
• Fixed the issue with inverted indexes for VARIANT type fields. ([fix](variant) fix index in variant #43375)
• Optimized local cache locality for inverted indexes to improve cache hit rates. ([fix](inverted index) inverted Index File Cache Queue Optimization #46518)
• Added the metric NumInvertedIndexRemoteIOTotal to query profiles for remote storage reads of inverted indexes. ([opt](profile) add index page profile for io #45675, [opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile #44863)

Others

• Fixed the crash issue of the ipv6_cidr_to_range function with special NULL data. ([fix](ip) fix ip nullable param without check #44700)

Permissions

• When granting CREATE_PRIV, the existence of the corresponding resource is no longer checked. ([enhance](auth)When authorization includes create, not check if resources exist #45125)
• Fixed the issue where queries on views with permissions could fail due to missing permissions for referenced tables in extreme scenarios. ([fix](auth)Fix the need for low-level table permissions when querying views in certain situations #44621)
• Resolved the issue where permission checks for use db did not distinguish between internal and external Catalogs. ([fix](auth) fix use database stmt access unauthorized catalog #45720)

[gavinchou]

行为变更

• 在 Audit log 中，drop table 和 drop database 语句保持 force 标志。（[fix](drop sql) add force in the tosql for drop table and drop database #43227）
• 导出数据至 Parquet/ORC 格式时，bitmap、quantile_state 和 hll 类型将以 Binary 格式导出。同时新增支持导出 jsonb 和 variant 类型，导出格式为 string。（[fix](Outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #44041）
  • 相关文档：[Export Overview - Apache Doris](https://doris.apache.org/docs/3.0/data-operate/export/export-overview)
• 将 Hudi JNI Scanner 从 Spark API 替换为 Hadoop API，以增强兼容性。用户可以通过设置会话变量 set hudi_jni_scanner=spark/hadoop 进行切换。（[fix](hudi) upgrade hudi to 0.15.0 #44267）
• 禁止在 Colocate 表中使用 auto bucket。（[fix](table) Disable create, alter auto bucket table with colocate #44396）
• 为 Catalog 增加 Paimon 缓存，不再进行实时数据查询。（[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911）
• 增大 max_broker_concurrency 的默认值，以提升 Broker Load 在大规模数据导入时的性能。（[performance](load) increase max_broker_concurrency to 100 #44929）
• 将 Auto Partition 分区的 storage medium 默认值修改为当前表的 storage medium 属性值，而非系统默认值。（[Bug](auto-partition) fix auto partition could set storage_medium properties #45955）
• 禁止在修改 Key 列的 Schema Change 执行期间进行列更新。（[Fix](partial update) abort partial update on shadow index's tablet when the including columns miss key columns on new schema #46347）
• 对于包含自增列的 Key 列表，支持在列更新时不提供自增列。（[opt](auto-inc) Allow to miss auto-increment column and other value columns in partial update #44528）
• FE ID 生成器策略切换为与物理时间相关的策略，ID 不再从 10000 开始。（[chore](conf) Set enable_advance_next_id=true by default #44790）
• 在存算分离模式下，Compaction 产生的 stale rowset 默认回收延迟时间减小至 1800 秒，以减少回收间隔。某些极端场景下可能会导致超大查询失败，如遇问题可按需调整。（[Enhancement](config) Modify cloud default stale rowset recycle time #45460）
• 在存算分离模式下禁用 show cache hotspot 语句，需直接访问系统表。（[chore](file cache) Disable show cache hotspot stmt #47332）
• 禁止删除系统创建的 admin 用户。（[fix](auth)Prohibit deleting admin user #44751）

改进

存储

• 优化 Routine Load 因 max_match_interval 设置过小导致任务频繁超时的问题。（[improve](routine load) introduce routine load task min timeout #46292）
• 提升 Broker Load 在导入多个压缩文件时的性能。（branch-3.0: [performance](load) fix broker load scan ranges for unsplittable files #43161 #43975）
• 增大 webserver_num_workers 的默认值以提升 Stream Load 性能。（branch-3.0: [opt](load) Increase the default value of webserver_num_workers #46304 #46593）
• 优化 Routine Load 导入任务在 BE 节点扩容时负载不均衡的问题。（branch-3.0: [improve](routine load) ensure load balance after scaling up BE nodes #44693 #44798）
• 优化 Routine Load 线程池使用，防止 Routine Load 超时失败影响查询。（branch-3.0: [fix](routine load) replace heavy work pool with routine load thread pool for metadata fetching #44907 #45039）

存算分离

• 加强 Meta-service 的稳定性和可观测性。（[improve][ms] Bvars add the FDB get_count_normalized indicator #44036, [opt](recycler) Improve robustness and observability  #45617, [Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers #45255, [opt](cloud) reset ms response message in case it is reused for retry #45068）
• 优化 File Cache，增加提前淘汰策略，减小持锁时间，提升查询性能。（[enhancement](cloud) file cache evict in advance #47473, [enhancement](cloud) add profile counter for file cache #45678, [fix](cloud) shorten cache lock held time and add metrics #47472）
• 优化 File Cache 初始化检查以及队列转换，提升稳定性。（[fix](cloud) explicit message when parse file_cache_path error #44004, [fix](cloud) serialize cache init to avoid unstable cache pick #44429, [fix](cloud) fix CHECK failed when transmit from non-TTL to normal #45057, [chore](file_cache) Set enbale_dump_error_file to false by default #47229）
• 优化 HDFS 数据回收速度。（[Fix](recycler) Fix recycler Hdfs accessor list all bug #46393）
• 优化超高频导入时 FE 获取计算组可能存在的性能问题。（[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() #47203）
• 优化存算分离主键表的若干导入相关参数，提升实时高并发导入的稳定性。（[fix](cloud-mow) Make delete bitmap cache expired time more reasonable #47295, [improve](cloud-mow)Make some mow calculating delete bitmap config more reasonable #46750, [fix](cloud-mow) Make some timeout about mow more reasonable #46365）

Lakehouse

• 支持读取 Hive Json 格式的表数据。（[feature](hive)support hive catalog read json table.  #43469）
  • 相关文档：[Text/CSV/JSON - Apache Doris](https://doris.apache.org/docs/dev/lakehouse/file-formats/text#json)
• 支持会话变量 enable_text_validate_utf8，可忽略 CSV 格式中的 UTF-8 编码检测。（[enchement](utf8)import enable_text_validate_utf8 session var #45537）
  • 相关文档：[Text/CSV/JSON - Apache Doris](https://doris.apache.org/docs/dev/lakehouse/file-formats/text#character-set)
• 将 Hudi 版本更新至 0.15，并优化 Hudi 表的查询规划性能。
• 优化 MaxCompute 分区表的读取性能。（[enchement](mc)Optimize reading of maxcompute partition tables. #45148）
• 优化在高过滤率情况下，Parquet 文件延迟物化的性能。（branch-2.1: [fix](parquet-reader) Fixed the issue of excessive scanning data in late materialization‌ case of parquet reader #46121 #46183）
• 支持 Parquet 复杂类型的延迟物化。（[opt](parquet-reader)Implement late materialization of parquet complex types. #44098）
• 优化 ORC 类型的谓词下推逻辑，支持更多谓词条件用于索引过滤。（[enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255）

异步物化视图

• 支持更多场景下的聚合上卷改写。（[opt](mtmv) Support any_value aggregate function rollup rewrite when exsits aggregate materialized view #44412）

查询优化器

• 优化分区裁剪性能。（[opt](nereids) improve prune partition with lots of in (xxx) #46261）
• 增加利用数据特征消除 group by key 的规则。（[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform #43391）
• 根据目标表的数据量自适应调整 Runtime Filter 的等待时间。（[feat](nereids)set runtime filter wait time according to table row count and table type #42640）
• 优化聚合下压连接的能力，以适应更多场景。（[opt](nereids) enhance PUSH_DOWN_AGG_THROUGH_JOIN_ONE_SIDE #43856, [opt](nereids) support pushing down aggr distinct through join #43380）
• 优化 Limit 下压聚合，以适应更多场景。（[opt](nereids) optimize push limit to agg #44042）

其他

• 优化 FE、BE、MS 进程启动脚本，使输出内容更明确。（[chore](script) fix start_fe.sh --version not work and MetaService scripts occur error in Debian GNU/Linux 11 (bullseye) #45610, [chore](version) Show binary version in metrics: fe be ms #45490, [chore](bash) optimize output information when doris_cloud startup  #45883）
• show tables 显示的表名大小写现在与 MySQL 行为一致。（[fix](show)show tables should be case insensitive when lowerCaseTableNames is 1 or 2. #46030）
• show index 支持任意目标表类型。（[opt](show) let all types table support show index #45861）
• information_schema.columns 支持显示默认值。（[improvement](information_schema)Support show default value in information_schema. #44849）
• information_schema.views 支持显示视图定义。（[improvement](information_schema)Show view definition in information_schema.views. #45857）
• 支持 MySQL 协议的 COM_RESET_CONNECTION 命令。（[improvement](mysql)Support mysql COM_RESET_CONNECTION command. #44747）

缺陷修复

存储

• 修复聚合表模型导入过程中可能出现的内存错误。（[bugfix](memtable) arena is freed early and will cause use after free #46997）
• 修复存算分离模式下 FE 主节点重启时导致 Routine Load offset 丢失的问题。（branch-3.0: [fix](cloud) fix routine load loss data when fe master node restart #46149 #46566）
• 修复存算模式下 FE Observer 节点在批量导入场景中的内存泄漏问题。（branch-3.0: [fix](cloud)(bulk load) fix memory leak in FE observer node #47074 #47244）
• 修复 Full Compaction 进行 Order Data Compaction 导致 Cumulative Point 回退的问题。（[Fix](full compaction) Full compaction should not do ordered data compaction #44359）
• 修复 Delete 操作可能导致 Tablet Compaction 短暂无法调度的问题。（[Enhancement](compaction) Do not set failure time when cumulative compaction dealing with delete rowset #43466）
• 修复多计算集群时，Schema Change 后 Tablet 状态不正确的问题。（[enhancement](meta) Sync tablet meta even if local state is not running #45821）
• 修复在有 sequence_type 的主键表上进行 Column Rename Schema Change 时可能报 NPE 错误的问题。（[Fix](schema change) Fix NPE when rename column on table which has sequence type column #46906）
• 数据正确性：修复主键表在部分列更新导入包含 DELETE SIGN 列时的正确性问题。（[Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194）
• 修复主键表 Publish 任务持续卡住时，FE 可能存在内存泄漏的问题。（[Fix](TPartitionVersionInfo) Fix duplicate TPartitionVersionInfo in PublishVersionTask.partitionVersionInfos #44846）

存算分离

• 修复 File Cache 可能导致缓存大小大于表数据大小的问题。（[fix](cloud) fix file cache potential leakage #46561, [fix](cloud) Support clean tablet file cache when tablet drop #46390）
• 修复数据上传至 5MB 边界值时可能导致上传失败的问题。（[fix](s3filewriter) Fix s3_write_buffer_size boundary issue #47333）
• 修复 Storage Vault 若干 alter 相关操作，增加更多参数检查，提升鲁棒性。（[fix](storage vault) Fix missing use_path_style when create storage vault #45155, [fix](vault) avoid encrypt twice when altering vault #45156, [fix](vault) fix CreateTableLikeStmt cannot work in stoarge vault mode #46625, [opt](storage vault) Check s3.root.path cannot be empty #47078, [fix](vault) Fix bugs about altering storage vault name #45685, [fix](vault) Fix creating storage vault failed with azure backend #46779）
• 修复因 Storage Vault 配置不当导致数据无法回收或回收缓慢的问题。（[fix](recycler) Fix premature exit recycling when there is an invalid storage vault #46798, [Fix](recycler) Fix retain inverted indexes in tmp rowset recycling #47536, [Fix](recycler) Delete again to double check when recycle tablet failed by some bugs #47475, [Fix](recycler) Fix recycler fail when dealing with rowset [0-1] #47324, [fix](recycler) Process self-defined domain names for s3 storage vault #45072）
• 修复回收过程中可能卡住导致数据无法及时回收的问题。（[fix](recycler) Fix CountdownEvent error and hang #45760）
• 修复存算分离下 MTTM-230 错误时未正确重试的问题。（[fix](cloud) Fix async mtmv job retry when meet -230 in cloud #47370, [fix](cloud) Fix cloud -230 retry not reset ctx state #47326）
• 修复存算分离模式下 Decommission BE 时，Group Commit WAL 未回放完成的问题。（[fix](cloud) Fix cloud decomission and check wal #47187）
• 修复超过 2GB 的 Tablet Meta 导致 MS 不可用的问题。（[fix](meta-service) Avoid rowset meta exceeds 2G result in protobuf fatal #44780）
• 数据正确性：修复存算分离主键表的两个重复 Key 问题。（[Fix](merge-on-write) Should update pending delete bitmap KVs in MS when no need to calc delete bitmaps in publish phase  #46039, [Fix](merge-on-write) Should clear GetDeleteBitmapUpdateLockResponse when geting delete bitmap update lock fail and retry #44975）
• 修复存算分离主键表在高频实时导入下，可能因 Delete Bitmap 过大导致 Base Compaction 持续失败的问题。（[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap #46969）
• 修改 Schema Change 在存算分离主键表上的一些错误重试逻辑，提高 Schema Change 的健壮性。（[fix](cloud-mow) schema change should retry when encouter TXN_CONFILCT in cloud mode #46748）

Lakehouse

Hive

• 修复无法查询 Spark 创建的 Hive 视图的问题。（branch-2.1: [fix](hive) support query hive view created by spark #43553）
• 修复无法正确读取某些 Hive Transaction 表的问题。（[fix](hive)fix hive insert only translaction table. #45753）
• 修复 Hive 表分区存在特殊字符时，无法进行正确分区裁剪的问题。（[fix](hive)fix hive catalog miss partition that have special characters. #42906）

Iceberg

• 修复在 Kerberos 认证环境下，无法创建 Iceberg 表的问题。（[feat](catalog)Support Pre-Execution Authentication for HMS Type Iceberg Catalog Operations.  #43445）
• 修复某些情况下，Iceberg 表存在 dangling delete 情况下，count(*) 查询不准确的问题。（[fix](iceberg)Fix count(*) error with dangling delete problem #44039）
• 修复某些情况下，Iceberg 表列名不匹配导致查询错误的问题。（[fix](iceberg)Bring field_id with parquet files And fix map type's key optional #44470）
• 修复某些情况下，Iceberg 表分区被修改后无法读取的问题。（[enchement](iceberg)support read iceberg partition evolution table. #45367）

Paimon

• 修复 Paimon Catalog 无法访问阿里云 OSS-HDFS 的问题。（[Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS #42585）

Hudi

• 修复某些情况下，Hudi 表分区裁剪失效的问题。（[fix](hudi)Add hudi catalog read partition table partition prune #44669）

JDBC

• 修复某些情况下，开启表名大小写不敏感功能后，使用 JDBC Catalog 无法获取表的问题。

MaxCompute

• 修复某些情况下，MaxCompute 表分区裁剪失效的问题。（[enchement](maxcompute)add mc catalog read partition table partition prune #44508）

其他

• 修复某些情况下，Export 任务导致 FE 内存泄漏的问题。（[fix](Export) fix a memory leak in the FE because of the ExportJob #44019）
• 修复某些情况下，无法使用 HTTPS 协议访问 S3 对象存储的问题。（[fix](s3) do not replace https scheme if specified #44242）
• 修复某些情况下，Kerberos 认证票据无法自动刷新的问题。（[feat](catalog)Replace HadoopUGI with HadoopKerberosAuthenticator to Support Kerberos Ticket Auto-Renewal #44916）
• 修复某些情况下，读取 Hadoop Block 压缩格式文件出错的问题。（[fix](hive) fix block decompressor bug #45289）
• 查询 ORC 格式的数据时，不再下推 CHAR 类型的谓词，以避免可能的结果错误。（[Fix](ORC) Not push down fixed char type in orc reader  #45484）

异步物化视图

• 修复极端场景下查询透明改写可能导致规划或结果错误的问题。（[fix](mtmv) Fix filter position different but same causing rewritten by materialized view fail #44575, [fix](mtmv) Fix mv is deleted in nested mv causing query err and fix some test #45744）
• 修复极端场景下异步物化视图调度可能多产生构建任务的问题。（[fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020, [Improve](mtmv) skip the generation of invalid task for refresh mtmv #46280）

查询优化器

• 修复部分表达式改写可能产生错误表达式的问题。（[fix](nereids) fix UnknownValue's reference in simplify range rule #44637 #44770, branch-3.0: [fix](nereids) fix compare with long min for simplify comparison rule #44920, branch-3.0: [fix](nereids) fix compare with date like overflow #45868 #45922, branch-3.0: [fix](nereids) fix compare with date like literal #45382 #45596）
• 修复偶现的 SQL Cache 结果错误问题。（[fix](cache) fix same sql return wrong result when switch database with use db and enable sql cache #44782, [fix](sql_cache) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') #44631, [fix](nereids) fix sql cache bug and some tests #46443, branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47266）
• 修复部分场景下 Limit 下压聚合算子可能导致错误结果的问题。（[fix](Nereids) set correct sort key for aggregate  #45369）
• 修复部分场景下延迟物化优化产生错误执行计划的问题。（[fix](nereids) support one phase DeferMaterializeTopN #45693, [fix](nereids) topN filter: use ObjectId as map key instead of Topn node  #46551）

查询执行

• 修复正则表达式和 like 函数在特殊字符时结果不正确的问题。（[fix](hyperscan) Fix hyper scan fall back to re2 #44547）
• 修复 SQL Cache 在切换 DB 时结果可能不正确的问题。（[fix](cache) fix same sql return wrong result when switch database with use db and enable sql cache #44782）
• 修复一系列 Arrow Flight 相关问题。（[fix](arrow-flight-sql) Fix query result is empty and not return query error message #45023, [fix](arrow-flight-sql) Fix Doris NULL column conversion to arrow batch #43929）
• 修复当 HashJoin 的 Hash 表超过 4G 时，部分情况下结果错误的问题。（[Bug](join) fix columnstr64's offset overflow on serialize_value_into_arena #46461）
• 修复 convert_to 函数在中文字符时溢出的问题。（[fix](mem) heap-buffer-overflow for function convert_to #46405）
• 修复 group by 带 Limit 时，在极端情况下结果可能出错的问题。（[Bug](fix) Fix topn agg limit may get error result in when refresh heap #47844）
• 修复访问某些系统表结果可能不正确的问题。（[fix](local shuffle) Set serial execution for schema scan operator #47498）
• 修复 percentile 函数可能导致系统崩溃的问题。（[Fix](bug) Percentile* func core when percent args is negative number #47068）
• 修复单表查询带 Limit 时性能退化的问题。（branch-3.0: [fix](scan) Fix scan with limit #46035 #46090）
• 修复 StDistanceSphere 和 StAngleSphere 函数导致系统崩溃的问题。（[Fix](function) fix coredump of function StDistanceSphere and StAngleSphere #45508）
• 修复 map_agg 结果错误的问题。（[Bug](map) fix wrong result on map_agg with streaming agg #40454）

半结构化数据管理

BloomFilter Index

• 修复 BloomFilter Index 参数过大导致的异常。（[fix](bf index) add ngram bf index validation in nereids index definition check #45780）
• 修复 BloomFilter Index 写入时内存占用过高的问题。（[opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833）
• 修复删除列时 BloomFilter Index 没有正确删除的问题。（[fix](bloom filter)Fix drop column with bloom filter index #44361, [fix](bloom filter)Fix drop column with bloom filter (#41369) #43378）

Inverted Index

• 修复倒排索引构建过程中偶发崩溃的问题。（[fix] (build index) fix build index coredump #43246）
• 修复倒排索引合并时，出现次数为 0 的词占用空间的问题。（[fix](index compaction)Skip writing terms with a doc frequency of 0 #43113）
• 避免 Index Size 统计出现超大异常值。（[fix](index size) discard index size when meta size is invalid #46549）
• 修复 VARIANT 类型字段的倒排索引异常。（[fix](variant) fix index in variant #43375）
• 优化倒排索引的本地缓存局部性，提高缓存命中率。（[fix](inverted index) inverted Index File Cache Queue Optimization #46518）
• 在查询 Profile 中增加倒排索引读远程存储的指标 NumInvertedIndexRemoteIOTotal。（[opt](profile) add index page profile for io #45675, [opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile #44863）

其他

• 修复 ipv6_cidr_to_range 函数在特殊 NULL 数据时崩溃的问题。（[fix](ip) fix ip nullable param without check #44700）

权限

• 赋予 CREATE_PRIV 时，不再检查对应资源是否存在。（[enhance](auth)When authorization includes create, not check if resources exist #45125）
• 修复在极端场景下，可能出现的查询有权限的视图，但报错没有视图中引用的表的权限的问题。（[fix](auth)Fix the need for low-level table permissions when querying views in certain situations #44621）
• 修复 use db 时检查权限时不区分内外 Catalog 的问题。（[fix](auth) fix use database stmt access unauthorized catalog #45720）

[gavinchou]

Thanks all who contribute to this release:

924060929 BePPPower BiteTheDDDDt CalvinKirs DarvenDuan FreeOnePlus Gabriel39 HappenLee JNSimba Jibing-Li LiBinfeng-01 MoanasDaddyXu Mryange SWJTU-ZhangLei Sjwhello TangSiyang2001 Thearas TsukiokaKogane Vallishp XLPE Yukang-Lian Yulei-Yang airborne12 amorynan biohazard4321 bobhan1 cambyzju cjj2010 csun5285 dataroaring deardeng eldenmoon englefly feiniaofeiafei felixwluo freemandealer gavinchou hello-stephen htyoung hubgeter hust-hhb jacktengg justfortaste kaijchen kaka11chen liaoxin01 liujiwen-up liutang123 luwei16 morningman morrySnow mrhhsg mymeiyi nsnhuang pingchunzhang py023 qidaye qzsee seawinde shuke987 sollhui starocean999 suxiaogang223 vinlee19 w41ter wangbo wuwenchi wyxxxcat xinyiZzz xy720 xyfsjq xzj7019 yagagagaga yiguolei yujun777 zclllyybb zddr zfr9527 zgxme zhangstar333 zhannngchen zhiqiang-hhhh zxealous zy-kkk zzzxl1993