Skip to content
This repository has been archived by the owner on Feb 6, 2024. It is now read-only.

Improve the cleaning mechanism of ShardNode #296

Open
ZuLiangWang opened this issue Dec 28, 2023 · 0 comments
Open

Improve the cleaning mechanism of ShardNode #296

ZuLiangWang opened this issue Dec 28, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@ZuLiangWang
Copy link
Contributor

ZuLiangWang commented Dec 28, 2023 •
edited
Loading

Describe this problem
We found that the failover mechanism of the HoraeDB cluster failed, and the shard was not migrated when the machine went down.

Steps to reproduce

  • Make the etcd root path configuration in HoraeDB and HoraeMeta inconsistent.
  • Shut down a HoraeDB node.

Additional Information

  • Add drop ShardNode api to deal with some extreme situations.
  • Add a new way to detect failed nodes, not only relying on etcd's lease event.
    1. Use a background thread to continuously detect failed nodes.
    2. Detect failed nodes through heartbeat.
@ZuLiangWang ZuLiangWang added the bug Something isn't working label Dec 28, 2023
@ZuLiangWang ZuLiangWang mentioned this issue Dec 28, 2023
Merged
ZuLiangWang added a commit that referenced this issue Dec 28, 2023
@ZuLiangWang
feat: add drop shard node api (#295)
a76d770 
## Rationale
In some extreme cases (for this situation, please refer to
#296), the shard
node cannot be cleaned normally, and we need to add an operation and
maintenance interface to complete manual cleaning.

## Detailed Changes
* Add drop shard node api.

## Test Plan
Pass CI.
ZuLiangWang added a commit to ZuLiangWang/ceresmeta that referenced this issue Jan 10, 2024
@ZuLiangWang
feat: add drop shard node api (apache#295)
463aa0a 
## Rationale
In some extreme cases (for this situation, please refer to
apache#296), the shard
node cannot be cleaned normally, and we need to add an operation and
maintenance interface to complete manual cleaning.

## Detailed Changes
* Add drop shard node api.

## Test Plan
Pass CI.
ShiKaiWi pushed a commit that referenced this issue Jan 10, 2024
@ZuLiangWang @ShiKaiWi
feat: add drop shard node api (#295)
7e5268a 
## Rationale
In some extreme cases (for this situation, please refer to
#296), the shard
node cannot be cleaned normally, and we need to add an operation and
maintenance interface to complete manual cleaning.

## Detailed Changes
* Add drop shard node api.

## Test Plan
Pass CI.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ZuLiangWang