[Enhancement](compaction) support parallel compaction for single tablet

[gitccl]

Proposed changes

Issue Number: close #18742

Problem summary

Basic ideas

In this pr, we add support for parallel cumulative compaction for single tablet, and base compaction still runs in single thread.
Firstly, we save all current running cumulative compaction tasks in an array named cumulative_compactions, and save current running base compaction task in base_compaction.
In the case of multiple cumulative compaction tasks running at the same time, the time and order of completion of the tasks is indeterminate. Therefore, the rowsets that can be compacted by next thread are split into multiple contiguous segments. Every time we want to choose rowsets to compact, we will choose a contiguous segments with maximum score. And in this choose process, we will skip rowsets with large size to avoid their participation in cumulative compaction.
We also change the behavior of update_cumulative_point. In update_cumulative_point, we will forward cumulative_point as far as possible.

The use of lock

Since the clone task and the compression task cannot be performed at the same time, we use shared_mutex to ensure that the execution of the parallel compression task and the clone task is serialized. We also add a mutex compaction_meta_lock to protect the meta data of the compaction such as cumulative_compactions, base_compaction and cumulative_point.
The following describes the detailed use of locks in each function

1. In Tablet::calc_compaction_score, we hold compaction_meta_lock and meta_lock, thus we can safely access cumulative_compactions, cumulative_point, etc.
2. In Tablet::prepare_compaction_and_calculate_permits, before call prepare_compact, we will hold cumulative_compact_meta_lock. Thus in prepare_compact, we can safely access cumulative_compactions, cumulative_point, etc. The reason we hold cumulative_compact_meta_lock before calling prepare_compact because we need to ensure that choosing the rowsets to compact and adding the compaction task to cumulative_compactions must be an atomic operation. It is the same for base compaction, we also need to get cumulative_compact_meta_lock first, and then check whether base_compaction is null, if it is not null, it means that base compaction is already running (e.g. triggered by http request).
3. In CumulativeCompaction::execute_compact_impl, we will try to get reader lock of cumulative_compaction_lock first, and if we can't get it, that means the tablet is under clone. After do_compaction, we will hold compaction_meta_lock and call update_cumulative_point to safely forward cumulative_point.
4. In EngineCloneTask::_finish_clone, in addition to acquiring base_compaction_lock, cumulative_compaction_lock(write lock), we also acquire the cumulative_compact_meta_lock to access current running compaction tasks and update their is_clone_occurred field.

Checklist(Required)

• Does it affect the original behavior
• Has unit tests been added
• Has document been added or modified
• Does it need to update dependencies
• Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[gitccl]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.12 seconds
stream load tsv: 421 seconds loaded 74807831229 Bytes, about 169 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230426164133_clickbench_pr_136006.html

[gitccl]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[gitccl]

run p0

[gitccl]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!