Skip to content

Commit e88b2d9

Browse files
committed
clusterer_controller: maintenance of another node, and the full level (CC-10b)
clusterer_controller:maintenance takes an optional node_id: the request is multicast to the cluster (MAINT_SET, sent three times - it is idempotent) and the node named applies it to itself, so it takes effect only while that node is reachable. A new level, full, keeps everything evicted does and in addition makes every other node disable it in its clusterer (set_state STATE_DISABLED): they take their link to it down and stop sending it replication; leaving full enables it again. A node never disables itself. MASTER_ALIVE carries a second bitmap, the nodes at the full level, after the live count; older builds do not read past the live count. A member's ALIVE carries its level as before, and a level this build does not know counts as evicted. cl_ctr_get_node_maint(cluster_id, node_id, var) reads any member's level; $cl_ctr_role and cl_ctr_get_node_role() report maintenance for a node in maintenance; $cl_ctr_maintenance, cl_ctr_set_maintenance() and the MI command accept full.
1 parent 3d47ed5 commit e88b2d9

10 files changed

Lines changed: 343 additions & 41 deletions

File tree

‎modules/clusterer_controller/README.md‎

Lines changed: 40 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -639,18 +639,22 @@ opensips-cli -x mi clusterer_controller:shtag_auto cluster_id=1
639639

640640
#### clusterer_controller:maintenance
641641

642-
Put THIS node in or out of maintenance, e.g. for a rolling upgrade. A node in maintenance (*evicted*) stays a member - it keeps its node_id, its BIN links and its place in *clusterer_controller:list_members* - but it is never master or backup and holds no sharing tag. If it is the master, it hands over at once: its last keepalive carries the new state, and every member elects the successor as soon as it reads it, without waiting for the master watchdog. A *clusterer_controller:shtag_force* pin is refused for a node in maintenance, and a pin already on a node that goes into maintenance is cleared (automatic allocation resumes). Only when every node in the election window is in maintenance does the cluster take one of them as master anyway - a cluster must have a master.
642+
Put this node - or, with *node_id*, any member - in or out of maintenance, e.g. for a rolling upgrade. A node in maintenance (*evicted* or *full*) stays a member - it keeps its node_id, its BIN links and its place in *clusterer_controller:list_members* - but it is never master or backup and holds no sharing tag. If it is the master, it hands over at once: its last keepalive carries the new state, and every member elects the successor as soon as it reads it, without waiting for the master watchdog. A *clusterer_controller:shtag_force* pin is refused for a node in maintenance, and a pin already on a node that goes into maintenance is cleared (automatic allocation resumes). Only when every node in the election window is in maintenance does the cluster take one of them as master anyway - a cluster must have a master.
643643

644-
The state is announced by the node itself and relayed by the master, so every node agrees on it and it survives a master failover. It is runtime-only: a restart brings the node back in service.
644+
At *full*, every other node also disables it in its *clusterer* (as *clusterer:set_status* would), so they take their link to it down and stop sending it replication; leaving *full* enables it again and *clusterer* resumes pinging it. A node never disables itself - its peers do.
645+
646+
The state is announced by the node itself and relayed by the master, so every node agrees on it and it survives a master failover. A request for another node is multicast to the cluster and that node applies it to itself, so it takes effect only while that node is reachable. The state is runtime-only: a restart brings the node back in service.
645647

646648
Parameters:
647649

648650
- *cluster_id* — the target cluster.
649-
- *state* — *evicted* (or *on*) to enter maintenance, *none* (or *off*) to leave it.
651+
- *state* — *evicted* (or *on*) or *full* to enter maintenance, *none* (or *off*) to leave it.
652+
- *node_id* (optional) — the member to act on; this node when omitted.
650653

651654
```bash title="clusterer_controller:maintenance usage"
652655
opensips-cli -x mi clusterer_controller:maintenance cluster_id=1 state=evicted
653-
opensips-cli -x mi clusterer_controller:maintenance cluster_id=1 state=none
656+
opensips-cli -x mi clusterer_controller:maintenance cluster_id=1 state=full node_id=3
657+
opensips-cli -x mi clusterer_controller:maintenance cluster_id=1 state=none node_id=3
654658
```
655659

656660
### Exported Pseudo-Variables
@@ -659,7 +663,7 @@ These read-only pseudo-variables expose live cluster state to the routing script
659663

660664
Each variable optionally takes a cluster id as its argument, e.g. *$cl_ctr_role(2)*. The bare form (*$cl_ctr_role*) resolves to the only configured cluster; when several clusters are defined the bare form returns NULL and logs a one-time warning, so the cluster must be named explicitly. An unknown cluster id, or a value that is not currently known (e.g. no master yet), returns NULL. All of them are read-only - assigning to them fails.
661665

662-
- **$cl_ctr_role** — this node's role in the cluster: *master*, *backup*, *member*, or *joining* (still authenticating / before the first election).
666+
- **$cl_ctr_role** — this node's role in the cluster: *master*, *backup*, *maintenance* (see *clusterer_controller:maintenance*), *member*, or *joining* (still authenticating / before the first election).
663667
- **$cl_ctr_is_master** — 1 if this node is the cluster master, 0 otherwise. A fast path for the most common check.
664668
- **$cl_ctr_master_ip** — IP of the current master (NULL if none is elected yet).
665669
- **$cl_ctr_backup_ip** — IP of the current backup / standby master (NULL if none).
@@ -668,7 +672,7 @@ Each variable optionally takes a cluster id as its argument, e.g. *$cl_ctr_role(
668672
- **$cl_ctr_members** — number of live members currently in the cluster.
669673
- **$cl_ctr_shtag_mode** — *auto* (tags follow the elected master) or *forced* (an operator pinned them with *clusterer_controller:shtag_force*).
670674
- **$cl_ctr_forced_node** — the *node_id* the active sharing tag is pinned to (NULL when in *auto* mode).
671-
- **$cl_ctr_maintenance** — this node's maintenance state: *none* or *evicted* (see *clusterer_controller:maintenance*).
675+
- **$cl_ctr_maintenance** — this node's maintenance state: *none*, *evicted* or *full* (see *clusterer_controller:maintenance*).
672676

673677
```opensips title="Using $cl_ctr_* in the script"
674678
# single cluster: run a periodic job only on the master
@@ -695,7 +699,7 @@ Meaning of the parameters is as follows:
695699

696700

697701
- *cluster_id* (int) - the cluster;
698-
- *state* (string) - *evicted* (or *on*) to enter maintenance, *none* (or *off*) to leave it.
702+
- *state* (string) - *evicted* (or *on*) or *full* to enter maintenance, *none* (or *off*) to leave it.
699703

700704

701705
Returns *1* on success, *-1* if the state is not recognised, the cluster is unknown or the controller worker is not running yet.
@@ -769,7 +773,7 @@ if (!cl_ctr_node_present(1, $var(peer)))
769773
#### cl_ctr_get_node_role(cluster_id, node_id, role_var)
770774

771775

772-
Writes a node's role - *master*, *backup* or *member* - into a variable.
776+
Writes a node's role - *master*, *backup*, *maintenance* or *member* - into a variable.
773777

774778

775779
Meaning of the parameters is as follows:
@@ -794,6 +798,34 @@ if (cl_ctr_get_node_role(1, 3, $var(role)))
794798
```
795799

796800

801+
#### cl_ctr_get_node_maint(cluster_id, node_id, maint_var)
802+
803+
804+
Writes a node's maintenance state - *none*, *evicted* or *full* - into a variable.
805+
806+
807+
Meaning of the parameters is as follows:
808+
809+
810+
- *cluster_id* (int) - the cluster;
811+
- *node_id* (int) - the node;
812+
- *maint_var* (var) - receives the state.
813+
814+
815+
Returns *1* on success, *-1* if the node is not a member.
816+
817+
818+
This function can be used from any route.
819+
820+
821+
```opensips title="cl_ctr_get_node_maint() usage"
822+
...
823+
if (cl_ctr_get_node_maint(1, 3, $var(m)) && $var(m) != "none")
824+
xlog("node 3 is in maintenance ($var(m))\n");
825+
...
826+
```
827+
828+
797829
#### cl_ctr_get_node_ip(cluster_id, node_id, ip_var)
798830

799831

@@ -1244,11 +1276,6 @@ Within one wire version, builds are kept compatible in both directions, so they
12441276

12451277
The following features are planned for future releases:
12461278

1247-
- **Node maintenance, the rest of it** — the *evicted* level is implemented (*clusterer_controller:maintenance*, *cl_ctr_set_maintenance()*, *$cl_ctr_maintenance*, *E_CL_CTR_MAINTENANCE*). Still planned:
1248-
1249-
- a *full* level that additionally marks the node down for clusterer consumers, so peers stop routing replication work to it (needs a clusterer hook to set a node's state);
1250-
- setting another node's state from any node (today a node sets its own);
1251-
- *$cl_ctr_node_maint(cluster_id, node_id)* for any peer, and a *maintenance* value of *$cl_ctr_role*.
12521279
- **Statistics** — module statistics (current role, member count, master changes, nodes joined/left, JOIN_REJECTs, decrypt failures, config mismatches, split-brain merges) exposed via *get_statistics* and monitoring exporters.
12531280
- **Events** — events raised on state transitions (became master, demoted, node joined/left, split-brain merged, config mismatch, authentication reject), named `E_CL_CTR_*` and consumable from an *event_route* or any event subscriber transport.
12541281
- **IPv6 multicast** — the control plane currently uses IPv4 multicast (groups in *224.0.0.0/4*). Add IPv6 multicast support (*ff00::/8* groups, *AF_INET6* sockets and membership) so the controller can run on IPv6-only or dual-stack deployments.

‎modules/clusterer_controller/cl_ctr.h‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,7 @@
211211
#include "../../pvar.h" /* pv_export_t - read-only $cl_ctr_* variables */
212212

213213
#include "../clusterer/clusterer_ctrl.h" /* set_my_identity, add_node, remove_node */
214+
#include "../clusterer/api.h" /* set_state() - the 'full' maintenance level */
214215

215216
#include <sys/timerfd.h> /* timerfd_create(), timerfd_settime() */
216217
#include "../../reactor_proc.h" /* reactor_proc_init/add_fd/loop */
@@ -349,6 +350,12 @@ static inline long long cl_ctr_now_ms(void)
349350
* the master relays every node's in MASTER_ALIVE. */
350351
#define CL_CTR_MAINT_NONE 0
351352
#define CL_CTR_MAINT_EVICTED 1
353+
/* full: evicted, and every other node marks it down in its own clusterer
354+
* (set_state STATE_DISABLED), so replication stops routing work to it */
355+
#define CL_CTR_MAINT_FULL 2
356+
/* any node -> group: put node [node_id 2B BE] into maintenance [state 1B];
357+
* only the node named applies it, to itself, and announces it as usual */
358+
#define CL_CTR_PKT_MAINT_SET 0x11
352359

353360
/* Consumer messaging (api.h): plaintext payload layout after type+seq is
354361
* [src_node_id u16 BE][chan_len u8][channel bytes][consumer payload].
@@ -1494,6 +1501,8 @@ extern char my_interface_buf[IF_NAMESIZE];
14941501
extern uint16_t my_node_id;
14951502
extern uint64_t cl_ctr_incarnation;
14961503
extern clusterer_ctrl_binds_t clctl;
1504+
extern struct clusterer_binds cl_api; /* the public clusterer API */
1505+
extern int cl_api_loaded;
14971506
extern int clctl_loaded;
14981507
extern int manage_shtags;
14991508
extern int consumer_retries;
@@ -1592,7 +1601,7 @@ int cl_ctr_on_retx_tfd(int fd, void *param, int was_timeout);
15921601

15931602
/* cl_ctr_peers.c */
15941603
int cl_ctr_find_peer(int cid, int nid, int *is_master, int *is_backup,
1595-
char *ipbuf, int ipbuf_sz);
1604+
int *maint, char *ipbuf, int ipbuf_sz);
15961605
unsigned int ip_to_num(const char *ip);
15971606
void cl_ctr_prune_stale(cl_ctr_cluster_t *cl);
15981607
void cl_ctr_apply_master_from_list_locked(const char *master_ip, cl_ctr_cluster_t *cl);
@@ -1630,11 +1639,19 @@ int cl_ctr_maint_parse(const char *s, int len);
16301639
void cl_ctr_maint_sync(cl_ctr_cluster_t *cl);
16311640
void cl_ctr_rpc_maint(int sender, void *param);
16321641
int cl_ctr_set_my_maint(cl_ctr_cluster_t *cl, int st);
1642+
int cl_ctr_set_node_maint(cl_ctr_cluster_t *cl, uint16_t node_id, int st);
16331643
void cl_ctr_build_maint_bitmap(cl_ctr_cluster_t *cl, unsigned char *bm);
16341644
int cl_ctr_apply_maint_bitmap(cl_ctr_cluster_t *cl, const unsigned char *bm,
1645+
const unsigned char *bm_full,
16351646
const char *master_ip);
1647+
void cl_ctr_build_maint_full_bitmap(cl_ctr_cluster_t *cl, unsigned char *bm);
16361648
void cl_ctr_raise_maint_event(cl_ctr_cluster_t *cl, int node_id, int st);
16371649
void cl_ctr_maint_peer_changed(cl_ctr_cluster_t *cl, uint16_t node_id, int st);
1650+
void cl_ctr_maint_changed(cl_ctr_cluster_t *cl, uint16_t node_id, int old_st,
1651+
int st);
1652+
int cl_ctr_send_maint_set(cl_ctr_cluster_t *cl, uint16_t node_id, int st);
1653+
void cl_ctr_handle_maint_set(const char *sender_ip, const char *payload,
1654+
int payload_len, cl_ctr_cluster_t *cl);
16381655
void cl_ctr_reelect_now(cl_ctr_cluster_t *cl, const char *why);
16391656
long long cl_ctr_master_timeout_ms(cl_ctr_cluster_t *cl);
16401657
void cl_ctr_arm_master_watchdog(cl_ctr_cluster_t *cl);

‎modules/clusterer_controller/cl_ctr_election.c‎

Lines changed: 98 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -538,12 +538,15 @@ void cl_ctr_request_rekey(cl_ctr_cluster_t *cl)
538538

539539
const char *cl_ctr_maint_name(int st)
540540
{
541-
return st == CL_CTR_MAINT_EVICTED ? "evicted" : "none";
541+
return st == CL_CTR_MAINT_FULL ? "full"
542+
: st == CL_CTR_MAINT_EVICTED ? "evicted" : "none";
542543
}
543544

544545
/* "evicted"/"on"/"1" -> CL_CTR_MAINT_EVICTED, "none"/"off"/"0" -> NONE, else -1 */
545546
int cl_ctr_maint_parse(const char *s, int len)
546547
{
548+
if (len == 4 && strncasecmp(s, "full", 4) == 0)
549+
return CL_CTR_MAINT_FULL;
547550
if ((len == 7 && strncasecmp(s, "evicted", 7) == 0) ||
548551
(len == 2 && strncasecmp(s, "on", 2) == 0) ||
549552
(len == 1 && s[0] == '1'))
@@ -603,19 +606,22 @@ void cl_ctr_reelect_now(cl_ctr_cluster_t *cl, const char *why)
603606
*/
604607
void cl_ctr_maint_sync(cl_ctr_cluster_t *cl)
605608
{
606-
int i, st, was_master = 0;
609+
int i, st, old_st, was_master = 0;
610+
uint16_t me = 0;
607611

608612
lock_start_write(cl->peers->lock);
609613
st = cl->peers->my_maint;
610614
if (st == cl->maint_adv) {
611615
lock_stop_write(cl->peers->lock);
612616
return;
613617
}
618+
old_st = cl->maint_adv;
614619
cl->maint_adv = (uint8_t)st;
615620
for (i = 0; i < cl->peers->count; i++)
616621
if (strcmp(cl->peers->entries[i].ip, my_ip) == 0) {
617622
cl->peers->entries[i].maint = (uint8_t)st;
618623
was_master = cl->peers->entries[i].is_master;
624+
me = cl->peers->entries[i].node_id;
619625
}
620626
lock_stop_write(cl->peers->lock);
621627

@@ -624,7 +630,7 @@ void cl_ctr_maint_sync(cl_ctr_cluster_t *cl)
624630
cl_ctr_maint_name(st),
625631
was_master && st != CL_CTR_MAINT_NONE
626632
? " - it is the master and hands over" : "");
627-
cl_ctr_raise_maint_event(cl, my_node_id, st);
633+
cl_ctr_maint_changed(cl, me, old_st, st);
628634
cl_ctr_send_alive(cl->sock, cl);
629635
if (was_master && st != CL_CTR_MAINT_NONE) {
630636
cl_ctr_send_master_alive(cl->sock, cl);
@@ -638,6 +644,55 @@ void cl_ctr_rpc_maint(int sender, void *param)
638644
cl_ctr_maint_sync((cl_ctr_cluster_t *)param);
639645
}
640646

647+
/* An operator's request, made on this node, for ANOTHER node's maintenance
648+
* state. The worker multicasts it (it owns the sequence space); the node
649+
* named applies it to itself (cl_ctr_handle_maint_set()). */
650+
typedef struct {
651+
cl_ctr_cluster_t *cl;
652+
uint16_t node_id;
653+
uint8_t st;
654+
} cl_ctr_maint_req_t;
655+
656+
static void cl_ctr_rpc_maint_remote(int sender, void *param)
657+
{
658+
cl_ctr_maint_req_t *r = (cl_ctr_maint_req_t *)param;
659+
660+
if (cl_ctr_send_maint_set(r->cl, r->node_id, r->st) < 0)
661+
LM_WARN("clusterer_controller: [cluster %d] could not send the request "
662+
"for node %u to go into maintenance state %s\n",
663+
r->cl->cluster_id, r->node_id, cl_ctr_maint_name(r->st));
664+
else
665+
LM_INFO("clusterer_controller: [cluster %d] asked node %u into "
666+
"maintenance state %s\n", r->cl->cluster_id, r->node_id,
667+
cl_ctr_maint_name(r->st));
668+
shm_free(r);
669+
}
670+
671+
/* Request node @node_id's maintenance state @st (MI, any process).
672+
* Returns 0, or -1 when the worker is not up yet or out of memory. */
673+
int cl_ctr_set_node_maint(cl_ctr_cluster_t *cl, uint16_t node_id, int st)
674+
{
675+
cl_ctr_maint_req_t *r;
676+
int proc_no;
677+
678+
lock_start_read(cl->peers->lock);
679+
proc_no = cl->peers->worker_proc_no;
680+
lock_stop_read(cl->peers->lock);
681+
if (proc_no < 0)
682+
return -1;
683+
r = shm_malloc(sizeof *r);
684+
if (!r)
685+
return -1;
686+
r->cl = cl;
687+
r->node_id = node_id;
688+
r->st = (uint8_t)st;
689+
if (ipc_send_rpc(proc_no, cl_ctr_rpc_maint_remote, r) < 0) {
690+
shm_free(r);
691+
return -1;
692+
}
693+
return 0;
694+
}
695+
641696
/*
642697
* cl_ctr_maint_peer_changed() - a peer announced a new maintenance state
643698
* (master path: heard in its ALIVE). Raises the event, and when it went
@@ -648,7 +703,6 @@ void cl_ctr_maint_peer_changed(cl_ctr_cluster_t *cl, uint16_t node_id, int st)
648703
{
649704
int cleared = 0, im;
650705

651-
cl_ctr_raise_maint_event(cl, node_id, st);
652706
if (st == CL_CTR_MAINT_NONE || node_id == 0)
653707
return;
654708
lock_start_write(cl->peers->lock);
@@ -733,3 +787,43 @@ void cl_ctr_isolation_round(cl_ctr_cluster_t *cl, int i_am_master)
733787
cl_ctr_apply_shtags(cl);
734788
}
735789
}
790+
791+
/*
792+
* cl_ctr_maint_changed() - node [node_id]'s maintenance state went from
793+
* @old_st to @st, as this node learned it (its own request, a member's ALIVE
794+
* on the master, the master's relay on a member). Raises the event and, for
795+
* another node entering or leaving the 'full' level, marks it down or up in
796+
* this node's clusterer, so replication stops (or resumes) routing work to
797+
* it. A node does not disable itself: its peers do. Called without the
798+
* peers lock held.
799+
*/
800+
void cl_ctr_maint_changed(cl_ctr_cluster_t *cl, uint16_t node_id, int old_st,
801+
int st)
802+
{
803+
uint16_t me = 0;
804+
int k;
805+
806+
cl_ctr_raise_maint_event(cl, node_id, st);
807+
if (node_id == 0 || !cl_api_loaded || !cl_api.set_state)
808+
return;
809+
/* this node's id in THIS cluster (it may differ per cluster): disabling
810+
* ourselves in clusterer would cut every link we have */
811+
lock_start_read(cl->peers->lock);
812+
for (k = 0; k < cl->peers->count; k++)
813+
if (strcmp(cl->peers->entries[k].ip, my_ip) == 0)
814+
me = cl->peers->entries[k].node_id;
815+
lock_stop_read(cl->peers->lock);
816+
if (node_id == me)
817+
return;
818+
if (st == CL_CTR_MAINT_FULL && old_st != CL_CTR_MAINT_FULL) {
819+
if (cl_api.set_state(cl->cluster_id, node_id, STATE_DISABLED) == 0)
820+
LM_INFO("clusterer_controller: [cluster %d] node %u is in full "
821+
"maintenance - disabled in clusterer here\n",
822+
cl->cluster_id, node_id);
823+
} else if (old_st == CL_CTR_MAINT_FULL && st != CL_CTR_MAINT_FULL) {
824+
if (cl_api.set_state(cl->cluster_id, node_id, STATE_ENABLED) == 0)
825+
LM_INFO("clusterer_controller: [cluster %d] node %u left full "
826+
"maintenance - enabled in clusterer here\n",
827+
cl->cluster_id, node_id);
828+
}
829+
}

0 commit comments

Comments
 (0)