Skip to content

Commit 0b3af68

Browse files
committed
clusterer_controller: join the multicast group on our own interface, not INADDR_ANY
cl_ctr_setup_socket() joined the peer-discovery multicast group with mreq.imr_interface.s_addr = INADDR_ANY, leaving the actual interface choice to the kernel's default-route selection instead of the interface the cluster was actually configured for (the "interface" modparam, already resolved into my_ip by cl_ctr_resolve_local_identity() before this runs). setsockopt(IP_ADD_MEMBERSHIP) succeeds either way - no error is logged - so this fails completely silently. Found live testing a real 2-node cluster for the first time (every prior deployment was single-node, where no peer discovery ever needed to actually cross the wire): two production hosts each joined 239.255.24.3 on their external, internet-facing interface (matching their default route) instead of the internal interface named by their own "interface=ens18" config, confirmed via 'ip maddr show' on both - the multicast membership was present on the external interface, absent on the internal one. Neither node's JOIN_REQ ever reached the other; both timed out waiting for a master that was never reachable and self-elected as sole node instead. Cross-checked against three already-working multi-node clusters elsewhere in the fleet - same INADDR_ANY code path, but their default route happens to already point out the same interface configured for clustering, so the kernel picked correctly there by coincidence. Nothing about this bug depends on production vs test/staging; it depends on default route vs configured interface agreeing, which they don't on multi-homed hosts where the default route goes out a non-clustering interface. Fix: bind the join to my_ip explicitly, matching what the send path already does a few lines below (local_if.s_addr = inet_addr(my_ip), used to tag loopback packets with the right source for self-detection) - the two were inconsistent with each other for no reason. Verified live: after the fix, two production nodes found each other, authenticated (KEY_GRANT) on the first attempt, and formed a correct master/backup pair, confirmed symmetrically from both sides' logs.
1 parent de9030a commit 0b3af68

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

‎modules/clusterer_controller/clusterer_controller.c‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2324,7 +2324,24 @@ static int cl_ctr_setup_socket(cl_ctr_cluster_t *cl)
23242324

23252325
memset(&mreq, 0, sizeof(mreq));
23262326
mreq.imr_multiaddr.s_addr = inet_addr(cl->multicast_address);
2327-
mreq.imr_interface.s_addr = htonl(INADDR_ANY);
2327+
/* Bind the join to our OWN resolved interface (my_ip, set by
2328+
* cl_ctr_resolve_local_identity() before this runs), not
2329+
* INADDR_ANY. INADDR_ANY leaves the actual interface choice to
2330+
* the kernel's default-route selection, which is NOT necessarily
2331+
* the interface this cluster was configured for - on a host whose
2332+
* default route goes out a different (e.g. external-facing)
2333+
* interface than the one named by the "interface" modparam, the
2334+
* join silently lands on the wrong interface: setsockopt still
2335+
* succeeds (no error logged), but peers on the intended interface
2336+
* never see it, so every node times out waiting for a master that
2337+
* was never reachable and self-elects instead. Found live: two
2338+
* production nodes each joined on their external interface
2339+
* (matching their default route) instead of the internal one the
2340+
* cluster was actually meant to run on, and neither ever saw the
2341+
* other. The send side already pins to my_ip a few lines below
2342+
* (local_if.s_addr = inet_addr(my_ip)) - this makes the join side
2343+
* consistent with it. */
2344+
mreq.imr_interface.s_addr = inet_addr(my_ip);
23282345

23292346
if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP,
23302347
&mreq, sizeof(mreq)) < 0) {

0 commit comments

Comments
 (0)