[DSIP-35][Alert] Refactor the alert thread model

[ruanwenjun]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

The alert server is designed as HA, and we use single thread in the active server to loop alert event and send alert event to remote e.g HTTP/Slack/Email...

The current thread model has some problems

• Once an alert is blocked due to the remote server network block, the whole alert server will be blocked.
• The throughput of the AlertServer depends on the main thread on the active server, and due to the event sending is io blocking operation, so the throughput is very small.

Design Detail

This issue aims to improve the throughput of the alert server.

In the new thread model, there will exist three kinds of threads in the alert server

• Event fetch thread
• Event loop thread
• Evend sending threads

Each kind of threads will not block others.

Compatibility, Deprecation, and Migration Plan

Compatibility with latest version.

Test Plan

Test by UT.

Code of Conduct

• I agree to follow this project's Code of Conduct