Repository navigation
Commit 0b3af68
committed
clusterer_controller: join the multicast group on our own interface, not INADDR_ANY
cl_ctr_setup_socket() joined the peer-discovery multicast group with
mreq.imr_interface.s_addr = INADDR_ANY, leaving the actual interface
choice to the kernel's default-route selection instead of the interface
the cluster was actually configured for (the "interface" modparam,
already resolved into my_ip by cl_ctr_resolve_local_identity() before
this runs). setsockopt(IP_ADD_MEMBERSHIP) succeeds either way - no
error is logged - so this fails completely silently.
Found live testing a real 2-node cluster for the first time (every
prior deployment was single-node, where no peer discovery ever needed
to actually cross the wire): two production hosts each joined
239.255.24.3 on their external, internet-facing interface (matching
their default route) instead of the internal interface named by their
own "interface=ens18" config, confirmed via 'ip maddr show' on both -
the multicast membership was present on the external interface,
absent on the internal one. Neither node's JOIN_REQ ever reached the
other; both timed out waiting for a master that was never reachable
and self-elected as sole node instead.
Cross-checked against three already-working multi-node clusters
elsewhere in the fleet - same INADDR_ANY code path, but their default
route happens to already point out the same interface configured for
clustering, so the kernel picked correctly there by coincidence.
Nothing about this bug depends on production vs test/staging; it
depends on default route vs configured interface agreeing, which they
don't on multi-homed hosts where the default route goes out a
non-clustering interface.
Fix: bind the join to my_ip explicitly, matching what the send path
already does a few lines below (local_if.s_addr = inet_addr(my_ip),
used to tag loopback packets with the right source for self-detection)
- the two were inconsistent with each other for no reason.
Verified live: after the fix, two production nodes found each other,
authenticated (KEY_GRANT) on the first attempt, and formed a correct
master/backup pair, confirmed symmetrically from both sides' logs.1 parent de9030a commit 0b3af68
1 file changed
Lines changed: 18 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2324 | 2324 | | |
2325 | 2325 | | |
2326 | 2326 | | |
2327 | | - | |
| 2327 | + | |
| 2328 | + | |
| 2329 | + | |
| 2330 | + | |
| 2331 | + | |
| 2332 | + | |
| 2333 | + | |
| 2334 | + | |
| 2335 | + | |
| 2336 | + | |
| 2337 | + | |
| 2338 | + | |
| 2339 | + | |
| 2340 | + | |
| 2341 | + | |
| 2342 | + | |
| 2343 | + | |
| 2344 | + | |
2328 | 2345 | | |
2329 | 2346 | | |
2330 | 2347 | | |
| |||
0 commit comments