Release Note 2.0.15

[xiaokang]

Previous Release Note 2.0.14

Thanks to our community users and developers, about 157 improvements and bug fixes have been made in Doris 2.0.15 version

You can access the full list through the GitHub link , with the key features and improvements highlighted below.

1 Behavior change

NA

2 New features

• Restore now supports deleting redundant tablets and partition options
  • [feature](restore) Support clean_tables/clean_partitions properties for restore job #39028

3 Improvement and optimizations

Stability

• Add a FE configuration abort_txn_after_lost_heartbeat_time_second for transaction abort time.
  • [improve](transaction) extend abort transaction time #28662
• Abort transactions after a BE loses heartbeat for over 1 minute instead of 5 seconds, to avoid overly sensitive transaction aborts.
  • [improvement](transaction) abort txn after be lost heartbeat over 1 min #22781
• Delay scheduling EOF tasks of routine load to avoid an excessive number of small transactions.
  • [improve](routine load) delay schedule EOF tasks to avoid too many small transactions #39975
• Prefer querying from online disk services to be more robust.
  • [improvement](query) prefer to chose tablet on alive disk #39467
• Skip checking newly inserted rows in non-strict mode partial updates if the row's delete sign is marked.
  • [Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322
• To prevent FE OOM, limit the number of tablets in backup tasks, with a default value of 300,000.
  • [chore](backup) limit the involved tablets in a backup job #39987

Performance

• Optimize slow column updates caused by concurrent column updates and compactions.
  • [opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487
• When a NullLiteral exists in a filter condition, it can now be folded into False and further converted to an EmptySet to reduce unnecessary data scanning and computation.
  • [improvement](Nereids) add where Null rule to create empty relation as where false #38135
• Improve performance of ORDER BY in permutation.
  • [Performance](opt) opt the order by performance in permutation #38985
• Improve the performance of string processing in inverted indexes.
  • [opt](inverted index) opt value extraction from column to string #37395

Optimizer and statistics

• Added support for statements beginning with a semicolon.
  • [opt](Nereids) support parse sql with semicolon at beginning #39399
• Polish aggregate function signature matching.
  • [opt](Nereids) polish aggregate function signature matching #39352
• Drop column statistics and trigger auto analyze after schema change.
  • [improvement](statistics)Drop column stats after schema change. #39101
• Support dropping cached stats using DROP CACHED STATS table_name.
  • [improvement](statistics)Support drop cached stats. #39367

Multi catalog and others

• Optimize JDBC Catalog refresh to reduce the frequency of client creation.
  • [improvement](jdbc catalog)Optimize JDBC Catalog refresh to reduce frequent client creation. #40261
• Fix thread leaks in JDBC Catalog under certain conditions.
  • [improvement](jdbc catalog) Force all resources to be closed in the close method #39423
• ARRAY MAP STRUCT types now support REPLACE_IF_NOT_NULL.
  • [improve](nested_types)support nested type with agg replace_if_not_null #38304
• Retry delete jobs for failures that are not DELETE_INVALID_XXX.
  • [opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834

Big Thanks

Thanks all who contribute to this release:

@924060929
@BePPPower
@BiteTheDDDDt
@CalvinKirs
@GoGoWen
@HappenLee
@Jibing-Li
@Johnnyssc
@LiBinfeng-01
@Mryange
@SWJTU-ZhangLei
@TangSiyang2001
@Toms1999
@Vallishp
@Yukang-Lian
@airborne12
@amorynan
@bobhan1
@cambyzju
@csun5285
@dataroaring
@eldenmoon
@englefly
@feiniaofeiafei
@hello-stephen
@htyoung
@hubgeter
@justfortaste
@liaoxin01
@liugddx
@liutang123
@luwei16
@mongo360
@morrySnow
@qidaye
@smallx
@sollhui
@starocean999
@w41ter
@xiaokang
@xzj7019
@yujun777
@zclllyybb
@zddr
@zhangstar333
@zhannngchen
@zy-kkk
@zzzxl1993

[w41ter]

In Apache Doris 2.0.15, we introduced a new parameter: max_backup_tablets_per_job, which is used to limit the maximum number of tablets involved in a backup, to avoid OOM issues caused by excessive tablets. See https://doris.apache.org/docs/2.0/admin-manual/config/fe-config?_highlight=fe#max_backup_tablets_per_job.

This might prevent some existing backup tasks from running. If you have such issues, please adjust the parameter's upper limit accordingly.