Skip to content

Commit 22cb916

Browse files
Lt-FlashLt-Flash
committed
clusterer_controller: zero-config HA clusterer control module
A control plane for the clusterer module: nodes discover each other over encrypted UDP multicast, elect a master, and are assigned their cluster identity at runtime, so an HA cluster needs no per-node id in the config and no database behind it. Native, controller-managed and hybrid topologies coexist - a cluster is controller-managed only if the clusterer declares it so via cluster_options. Crypto is libsodium-only, no fallback: XChaCha20-Poly1305 for the payload AEAD, Argon2id as the bootstrap KDF, and Noise_NNpsk0 for the join handshake, with X25519 / HKDF-SHA256 / RNG underneath. libsodium is linked dynamically (distro static libs are usually non-PIC), so target hosts need the runtime package. The module is added to exclude_modules in Makefile.conf.template, like the other modules with external-lib dependencies - enable it via include_modules - and libsodium-dev joins the CI apt requirements. Membership and liveness: - master election, with the invariant "MASTER_ALIVE keepalive armed <=> I am the elected master", which is what stops a node demoted purely by election from continuing to broadcast and flapping between two masters; - master-mediated ALIVE, so liveness costs O(N) messages per round rather than every node pinging every other (O(N^2)); - a membership digest carried in MASTER_ALIVE, with a RESYNC repair path when a node's view diverges from the master's; - the join handshake is ACKed and retransmitted under a bounded budget, and both the KEY_GRANT and the join-time state snapshot are unicast to the joining node instead of broadcast to everyone. Several controller clusters can share one BIN socket - each has its own multicast group, shared-port unicast is routed by cluster_id, and multicast is used as the fallback when two clusters collide on a port. Hardening: the join path is flood-DoS rate-limited and its inputs are validated and bounds-checked (including the Noise msg2 decrypt output and the zero-length cipherstate passthrough). Decrypt failures are classified rather than logged uniformly - a bootstrap-key failure means a wrong password, a foreign cluster or tampering and warns, while a transient session-key mismatch during a rekey or a split-brain heal is expected and stays at DBG. The split-brain defer budget resets on a fresh higher-IP JOIN_REQ, so a lower-IP node waits for a live higher-IP peer instead of self-promoting, bounded by CL_CTR_JOIN_DEFER_HARDMAX. Multicast is sent and joined on this node's own interface rather than INADDR_ANY, which is what makes discovery work on a multi-homed host where the default route does not carry the cluster network. Validated end-to-end on a 3-node cluster: controller / native / hybrid modes, master failover and election stability (staggered and simultaneous starts both converge to a single stable master), multiple controller clusters on one BIN socket, the MI surface and its error paths, buffer overrun/underrun fuzzing, wrong-password rejection and config mismatch. Ships with the admin guide, the HA test appendix, the generated README and a join-rejection test.
1 parent 1798c83 commit 22cb916

10 files changed

Lines changed: 10762 additions & 1 deletion

‎Makefile.conf.template‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@
7676
#uuid= UUID generator | uuid-dev
7777

7878
# Modules omitted from the default build, generally due to external dependencies.
79-
exclude_modules?= aaa_diameter aaa_radius auth_jwt auth_web3 b2b_logic_xml cachedb_cassandra cachedb_couchbase cachedb_dynamodb cachedb_memcached cachedb_mongodb cachedb_redis carrierroute cgrates compression cpl_c db_berkeley db_http db_mysql db_oracle db_perlvdb db_postgres db_sqlite db_unixodbc dialplan emergency event_rabbitmq event_kafka event_sqs h350 httpd http2d identity jabber json launch_darkly ldap lua mi_xmlrpc mmgeoip opentelemetry osp perl pi_http presence presence_dialoginfo presence_mwi presence_reginfo presence_xml presence_dfks proto_ipsec proto_sctp proto_tls proto_wss pua pua_bla pua_dialoginfo pua_mi pua_reginfo pua_usrloc pua_xmpp python regex rabbitmq_consumer rest_client rls rtp.io siprec sngtc snmpstats stir_shaken tls_mgm tls_openssl tls_wolfssl uuid xcap xcap_client xml xmpp
79+
exclude_modules?= aaa_diameter aaa_radius auth_jwt auth_web3 b2b_logic_xml cachedb_cassandra cachedb_couchbase cachedb_dynamodb cachedb_memcached cachedb_mongodb cachedb_redis carrierroute cgrates clusterer_controller compression cpl_c db_berkeley db_http db_mysql db_oracle db_perlvdb db_postgres db_sqlite db_unixodbc dialplan emergency event_rabbitmq event_kafka event_sqs h350 httpd http2d identity jabber json launch_darkly ldap lua mi_xmlrpc mmgeoip opentelemetry osp perl pi_http presence presence_dialoginfo presence_mwi presence_reginfo presence_xml presence_dfks proto_ipsec proto_sctp proto_tls proto_wss pua pua_bla pua_dialoginfo pua_mi pua_reginfo pua_usrloc pua_xmpp python regex rabbitmq_consumer rest_client rls rtp.io siprec sngtc snmpstats stir_shaken tls_mgm tls_openssl tls_wolfssl uuid xcap xcap_client xml xmpp
8080

8181
# Modules forced into the build, even when also listed in exclude_modules.
8282
include_modules?=

‎modules/clusterer_controller/Makefile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# WARNING: do not run this directly, it should be run by the master Makefile
2+
3+
include ../../Makefile.defs
4+
auto_gen=
5+
NAME= clusterer_controller.so
6+
7+
# libsodium is a hard requirement: it provides the payload AEAD
8+
# (XChaCha20-Poly1305), the bootstrap KDF (Argon2id), the Noise_NNpsk0 join
9+
# handshake, and X25519 / HKDF-SHA256 / RNG. It is linked dynamically (distro
10+
# static libs are usually non-PIC), so target hosts need the libsodium runtime
11+
# package (e.g. `apt install libsodium23`).
12+
SODIUM_EXISTS := $(shell pkg-config --exists libsodium 2>/dev/null && echo yes)
13+
ifeq ($(SODIUM_EXISTS),yes)
14+
DEFS += $(shell pkg-config --cflags libsodium)
15+
LIBS += $(shell pkg-config --libs libsodium)
16+
$(info clusterer_controller: libsodium found -> XChaCha20-Poly1305 + Argon2id + Noise_NNpsk0)
17+
else
18+
$(error clusterer_controller requires libsodium (install libsodium-dev / libsodium-devel))
19+
endif
20+
21+
include ../../Makefile.modules

‎modules/clusterer_controller/README‎

Lines changed: 1752 additions & 0 deletions
Large diffs are not rendered by default.

‎modules/clusterer_controller/clusterer_controller.c‎

Lines changed: 6609 additions & 0 deletions
Large diffs are not rendered by default.

‎modules/clusterer_controller/doc/clusterer_controller.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<?xml version="1.0" encoding='UTF-8'?>
2+
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3+
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
4+
<!ENTITY admin SYSTEM "clusterer_controller_admin.xml">
5+
<!ENTITY tests SYSTEM "clusterer_controller_tests.xml">
6+
<!ENTITY contrib SYSTEM "contributors.xml">
7+
<!-- Include general documentation entities -->
8+
<!ENTITY % docentities SYSTEM "../../../docs/entities.xml">
9+
%docentities;
10+
]>
11+
<book>
12+
<bookinfo>
13+
<title>CLUSTERER_CONTROLLER Module</title>
14+
<productname class="trade">&osipsname;</productname>
15+
</bookinfo>
16+
<toc></toc>
17+
&admin;
18+
&tests;
19+
&contrib;
20+
&docCopyrights;
21+
<para>&copyright; 2026 VoIPLine Telecom</para>
22+
</book>

‎modules/clusterer_controller/doc/clusterer_controller_admin.xml‎

Lines changed: 1931 additions & 0 deletions
Large diffs are not rendered by default.

‎modules/clusterer_controller/doc/clusterer_controller_tests.xml‎

Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
<!-- HA Behaviour Test Appendix -->
2+
3+
<appendix id="ha-tests" xreflabel="HA Behaviour Tests">
4+
<title>HA Behaviour Tests</title>
5+
<para>
6+
The following tests were performed on a three-node cluster
7+
(nodes <emphasis>A</emphasis>=10.22.23.191,
8+
<emphasis>B</emphasis>=10.22.23.192,
9+
<emphasis>C</emphasis>=10.22.23.193) to verify correct failover,
10+
tag stability, and no-steal-on-join behaviour. The sharing tag
11+
under test is <varname>vip1</varname> in cluster 1. All nodes run
12+
with <varname>manage_shtags=1</varname>.
13+
</para>
14+
<para>
15+
Tag state was queried after each operation via:
16+
<programlisting format="linespecific">opensips-cli -x mi clusterer_list_shtags</programlisting>
17+
</para>
18+
19+
<section id="ha-test-baseline">
20+
<title>Baseline</title>
21+
<para>
22+
All three nodes running. B holds the active tag; A and C are
23+
backup.
24+
</para>
25+
<programlisting format="linespecific">
26+
A (10.22.23.191) svc=active tag=backup
27+
B (10.22.23.192) svc=active tag=active
28+
C (10.22.23.193) svc=active tag=backup
29+
</programlisting>
30+
</section>
31+
32+
<section id="ha-test-1">
33+
<title>Test 1 — Stop the active node</title>
34+
<para>
35+
B (active) is stopped. The remaining nodes must elect a new
36+
active holder. B must rejoin as backup and must not steal the
37+
tag from whichever node became active.
38+
</para>
39+
<programlisting format="linespecific">
40+
# Stop B
41+
A svc=active tag=backup
42+
B svc=inactive tag=(down)
43+
C svc=active tag=active &lt;-- C promoted
44+
45+
# Start B
46+
A svc=active tag=backup
47+
B svc=active tag=backup &lt;-- rejoined as backup
48+
C svc=active tag=active &lt;-- C retains active
49+
</programlisting>
50+
<para>
51+
<emphasis role="bold">Result: PASS.</emphasis>
52+
Failover within the dead-node detection window; rejoining node
53+
did not steal the active tag.
54+
</para>
55+
</section>
56+
57+
<section id="ha-test-2">
58+
<title>Test 2 — Stop a backup node</title>
59+
<para>
60+
A (backup) is stopped. The active tag must remain on C without
61+
any transition. A must rejoin as backup.
62+
</para>
63+
<programlisting format="linespecific">
64+
# Stop A
65+
A svc=inactive tag=(down)
66+
B svc=active tag=backup
67+
C svc=active tag=active &lt;-- unchanged
68+
69+
# Start A
70+
A svc=active tag=backup &lt;-- rejoined as backup
71+
B svc=active tag=backup
72+
C svc=active tag=active &lt;-- still active
73+
</programlisting>
74+
<para>
75+
<emphasis role="bold">Result: PASS.</emphasis>
76+
Removing a backup node causes no tag movement; rejoining node
77+
started in backup state.
78+
</para>
79+
</section>
80+
81+
<section id="ha-test-3">
82+
<title>Test 3 — Stop both backup nodes</title>
83+
<para>
84+
A and B (both backup) are stopped simultaneously. The lone
85+
remaining node C must retain the active tag. A and B must
86+
rejoin as backup.
87+
</para>
88+
<programlisting format="linespecific">
89+
# Stop A and B
90+
A svc=inactive tag=(down)
91+
B svc=inactive tag=(down)
92+
C svc=active tag=active &lt;-- unchanged, lone node
93+
94+
# Start A, then B
95+
A svc=active tag=backup &lt;-- rejoined as backup
96+
B svc=active tag=backup &lt;-- rejoined as backup
97+
C svc=active tag=active &lt;-- still active
98+
</programlisting>
99+
<para>
100+
<emphasis role="bold">Result: PASS.</emphasis>
101+
Active node remained stable while running alone; both rejoining
102+
nodes came up in backup state.
103+
</para>
104+
</section>
105+
106+
<section id="ha-test-4">
107+
<title>Test 4 — Stop active node and one backup</title>
108+
<para>
109+
B (active) and C (backup) are stopped. The sole remaining node
110+
A must become active. B and C must rejoin as backup.
111+
</para>
112+
<programlisting format="linespecific">
113+
# Stop B and C
114+
A svc=active tag=active &lt;-- A promoted, now lone node
115+
B svc=inactive tag=(down)
116+
C svc=inactive tag=(down)
117+
118+
# Start B
119+
A svc=active tag=active &lt;-- retains active
120+
B svc=active tag=backup &lt;-- rejoined as backup
121+
C svc=inactive tag=(down)
122+
123+
# Start C
124+
A svc=active tag=active &lt;-- retains active
125+
B svc=active tag=backup
126+
C svc=active tag=backup &lt;-- rejoined as backup
127+
</programlisting>
128+
<para>
129+
<emphasis role="bold">Result: PASS.</emphasis>
130+
The surviving node correctly claimed the active tag; each
131+
rejoining node started in backup state without challenging the
132+
active holder.
133+
</para>
134+
</section>
135+
136+
<section id="ha-test-5">
137+
<title>Test 5 — Full cluster restart</title>
138+
<para>
139+
All three nodes are stopped (full outage). Nodes are then
140+
started one at a time. The first node up must self-elect as
141+
active (no peers available to sync from). Subsequent nodes must
142+
join as backup and must not steal the active tag.
143+
</para>
144+
<programlisting format="linespecific">
145+
# All stopped
146+
A svc=inactive tag=(down)
147+
B svc=inactive tag=(down)
148+
C svc=inactive tag=(down)
149+
150+
# Start A first
151+
A svc=active tag=active &lt;-- first/lone seed, self-synced
152+
B svc=inactive tag=(down)
153+
C svc=inactive tag=(down)
154+
155+
# Start B
156+
A svc=active tag=active &lt;-- retains active
157+
B svc=active tag=backup &lt;-- joined as backup, did not steal
158+
C svc=inactive tag=(down)
159+
160+
# Start C
161+
A svc=active tag=active &lt;-- retains active
162+
B svc=active tag=backup
163+
C svc=active tag=backup &lt;-- joined as backup, did not steal
164+
</programlisting>
165+
<para>
166+
<emphasis role="bold">Result: PASS.</emphasis>
167+
The first node to start elected itself active and no spurious
168+
sync errors were logged. Each subsequent node joined as backup
169+
without challenging the active holder.
170+
</para>
171+
</section>
172+
173+
<section id="ha-test-6">
174+
<title>Test 6 — Multiple clusters over one BIN socket</title>
175+
<para>
176+
Two controller clusters (<varname>id=1</varname> and
177+
<varname>id=2</varname>) are configured on all three nodes. Each
178+
cluster has its own multicast endpoint (a distinct port is required -
179+
two clusters on the same multicast address:port are rejected at
180+
startup with <emphasis>duplicate multicast</emphasis>), but both
181+
advertise the <emphasis>same</emphasis> BIN socket
182+
(<varname>bin:IP:3857</varname>). Each cluster must form independently
183+
and its replication must stay isolated over the shared socket.
184+
</para>
185+
<programlisting format="linespecific">
186+
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
187+
modparam("clusterer", "cluster_options", "cluster_id=2, use_controller=1")
188+
modparam("clusterer_controller", "cluster",
189+
"id=1,multicast=239.0.90.1:3333,bin_socket=bin:IP:3857")
190+
modparam("clusterer_controller", "cluster",
191+
"id=2,multicast=239.0.90.1:3334,bin_socket=bin:IP:3857")
192+
193+
# both clusters converge, each with its own master/backup/member roles;
194+
# clusterer_list shows cluster 1 and cluster 2 both using bin:IP:3857
195+
</programlisting>
196+
<para>
197+
<emphasis role="bold">Result: PASS.</emphasis>
198+
Both clusters formed independently (each <literal>member_count=3</literal>
199+
with consistent roles), the BIN links of both were <literal>Up</literal>
200+
over the single shared socket, and there were zero decrypt / cross-talk /
201+
foreign-cluster errors on any node - the <varname>cluster_id</varname> in
202+
each BIN packet keeps the two clusters' replication traffic separate. A
203+
control test confirmed that configuring both clusters on the same
204+
multicast address:port is refused at startup.
205+
</para>
206+
</section>
207+
208+
<section id="ha-test-summary">
209+
<title>Summary</title>
210+
<informaltable frame="all">
211+
<tgroup cols="3">
212+
<colspec colwidth="1*"/>
213+
<colspec colwidth="3*"/>
214+
<colspec colwidth="1*"/>
215+
<thead>
216+
<row>
217+
<entry>Test</entry>
218+
<entry>Scenario</entry>
219+
<entry>Result</entry>
220+
</row>
221+
</thead>
222+
<tbody>
223+
<row>
224+
<entry>1</entry>
225+
<entry>Stop active node; rejoin</entry>
226+
<entry>PASS</entry>
227+
</row>
228+
<row>
229+
<entry>2</entry>
230+
<entry>Stop backup node; rejoin</entry>
231+
<entry>PASS</entry>
232+
</row>
233+
<row>
234+
<entry>3</entry>
235+
<entry>Stop both backups simultaneously; rejoin</entry>
236+
<entry>PASS</entry>
237+
</row>
238+
<row>
239+
<entry>4</entry>
240+
<entry>Stop active + one backup; rejoin</entry>
241+
<entry>PASS</entry>
242+
</row>
243+
<row>
244+
<entry>5</entry>
245+
<entry>Full cluster restart; sequential startup</entry>
246+
<entry>PASS</entry>
247+
</row>
248+
<row>
249+
<entry>6</entry>
250+
<entry>Two clusters sharing one BIN socket (distinct multicast)</entry>
251+
<entry>PASS</entry>
252+
</row>
253+
</tbody>
254+
</tgroup>
255+
</informaltable>
256+
<para>
257+
In all five failover tests (1–5): exactly one node held the active
258+
sharing tag at all times (including during the failure window), and no
259+
rejoining node stole the active tag from the current holder. Test 6
260+
additionally confirmed that two clusters can share a single BIN socket
261+
with fully isolated replication.
262+
</para>
263+
</section>
264+
265+
</appendix>

‎modules/clusterer_controller/doc/contributors.xml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
<chapter>
2+
<title>Contributors</title>
3+
<section id="contributors" xreflabel="Contributors">
4+
<title>Contributors</title>
5+
<para>
6+
Definition, design and implementation of this module was made by:
7+
<itemizedlist>
8+
<listitem>
9+
<para>Yury Kirsanov — VoIPLine Telecom</para>
10+
</listitem>
11+
</itemizedlist>
12+
</para>
13+
</section>
14+
<section id="doc_contributors" xreflabel="Documentation Contributors">
15+
<title>Documentation Contributors</title>
16+
<para>
17+
Documentation was written by:
18+
<itemizedlist>
19+
<listitem>
20+
<para>Yury Kirsanov — VoIPLine Telecom</para>
21+
</listitem>
22+
</itemizedlist>
23+
</para>
24+
</section>
25+
</chapter>

0 commit comments

Comments
 (0)