[Lt-Flash]

Summary

This PR adds clusterer_controller, a contrib module that is clusterer with one extra ability: the nodes of a cluster find each other on their own. Give every node the same multicast group, cluster id and password, and they elect a master, which gives each node its node id, tells the others its BIN address and moves the sharing tags when a node goes away. The README has the details and an overview diagram, and doc/benchmarks.md the measurements.

Details

• No list of nodes to maintain. Every node runs the same configuration; a node is added, replaced or re-addressed without touching the others.
• A drop-in for clusterer. It is loaded instead of clusterer and keeps its name, parameters, MI commands and API, so dialog, usrloc, tm and the other modules that use clusterer work with it. The controller is switched on per cluster (use_controller=1); every other cluster behaves exactly as with clusterer.
• Sharing tags look after themselves. The master decides which node holds the active tag and hands it over when that node goes away.
• Large clusters form quickly. In my tests 64 nodes started at once formed in about two minutes, and 128 in about four and a half.
• Cheaper, sturdier replication. A module broadcast goes out as one multicast datagram instead of a TCP send to every node: 19% to 23% less cluster CPU than over TCP on a quiet network, and at 10% packet loss every dialog and contact still reached every node, where over TCP some did not.
• Encrypted control traffic. The controller's packets and its multicast broadcasts are encrypted and authenticated with keys derived from the cluster's password.

Solution

In a controller-managed cluster:

• Discovery and election. Every member multicasts a heartbeat each second, and the members elect a master.
• Admission. The master admits joiners a few at a time and gives each its node id. A joiner stays connecting - no role, no sharing tag - until its BIN links to every member are up both ways; one still connecting after 60 s is parked, so the queue moves on.
• Liveness. To the modules a node is up while it is heard and its link is up. There are no periodic pings, link-state floods or relaying.
• Broadcasts over multicast. Broadcasts are numbered, fragmented when large, and sent again to a node that asks; a TCP message from the same sender goes after the broadcasts sent before it, and a loss that cannot be repaired makes the modules that sync at startup sync again from the sender.
• One master. A joiner never founds a second cluster, a healed split keeps one master, and a master its members cannot hear steps down.

Compatibility

• Opt-in and not in the default build; clusters without the controller work exactly as before. It needs Linux, libsodium and IPv4 multicast between the nodes.
• Controller-managed clusters use plain BIN for now; BINS may come later.
• One small tm commit: tm reads the node id for the anycast Via cid per request instead of once at startup, as the controller assigns it at runtime. With clusterer nothing differs.

Closing issues

None.

[razvancrainea]

Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review.

[razvancrainea]

Hi, @Lt-Flash !

Once again, thank you for your contribution, it is truly appreciated. This is a very interesting topic for us!
For now, we decided to merge this module under the new modules-contrib/ directory, but long term plans would be to move it towards the main modules directory; therefore, we would like to prepare these changes for that moment! Do you think it is possible to copy the entire clusterer module under the new modules-contrib/ directory under a new name (it may be clusterer_controller), and then embed all the changes you made in this PR in this module. The goal is at the end to have one single module, that is close to clusterer as code base, but has these new features on top of them. And, of course, ideally make these features tunable (on/off).
This way we would have only one module in modules-contrib that is an enhancement of clusterer, that we will soon be able to move in the modules directory as a whole.
Do you think this is doable?

Best regards,
Răzvan

[Lt-Flash]

Hi Răzvan,

Thank you - that is great news, and I am glad the topic is interesting for you! Yes, it is doable, and I like the direction: a single module in modules-contrib/, adding the extra controller functionality to clusterer, will also be simpler than the current split.

To make sure I understood you correctly, this is what I plan:

• copy modules/clusterer as it is into modules-contrib/clusterer_controller as the first commit, so the rest of the series reads as a diff against clusterer;
• build the controller into that copy, so the module is clusterer itself with the controller added: clusterer's code changed where the controller needs it, the controller's own code in new files, and no changes to modules/clusterer. Being a separate module, it needs no build switches: the stock clusterer stays the default in the build, and the new one is selected like any other contrib module;
• keep it clusterer-compatible: the same parameter names, the same API (every module that uses clusterer works with it unchanged) and the same BIN protocol. The controller is switched on per cluster with use_controller=1; without it, a cluster behaves exactly as it does with clusterer, and the controller's own features keep their individual switches.

The module would be loaded instead of clusterer, not next to it.

Two questions:

1. tm reads the node id once at startup; with ids assigned at runtime it has to read it per request - this PR already does that (a small change, harmless with the stock clusterer). It lives in modules/tm, so it cannot move into the contrib module - would you prefer it as a separate small PR, or kept in this one?
2. Shall I force-push this PR with the new layout, or open a new one?

Thank you again for your time and for the support!

[Lt-Flash]

Hi @razvancrainea,

Thank you again for accepting the module! I have updated this PR as you asked: clusterer_controller is now a copy of clusterer under modules-contrib/, with the controller's features added on top and switched on per cluster, and modules/clusterer stays untouched.

Two questions, as the PR still has a few changes outside the module, each in its own commit:

1. tm now reads the node id per request instead of once at startup (needed when ids are assigned at runtime, harmless with the stock clusterer). Would you prefer it in a separate PR?
2. A few small core and packaging fixes the contrib module needed. Shall I move them into their own PR as well?

Happy to split either of them out - please let me know if you would like anything else changed.

[Lt-Flash]

A short note on why this update reworks clusterer_controller this much.

In my tests the version in this PR could not form a large cluster: 32 nodes started at once on one VM never settled, and even with the master admitting joiners a few at a time, clusterer's ping and link-state layer flooded itself at 64 nodes and the cluster did not form in any of three runs.

So controller-managed clusters now get an admission step, a multicast heartbeat in place of clusterer's pings, link-state floods and relaying, and module broadcasts over multicast. With that, 64 nodes started at once form in about two minutes. Clusters without use_controller are not affected; the details and numbers are in the updated description.

[Lt-Flash]

I have pushed an update: still the same 6 commits, and only the last one, the module itself, changed.

New tests and a second round of measurements found these, now fixed:

• After a split healed, a node that had elected itself while cut off - keeping its term, as nobody confirmed the old master gone - could win the tie on its higher IP and take over the cluster and its sharing tags. At the same member count and term, a mastership that raised the term now wins.
• With a cluster id such as 741, the tag of a module broadcast overflowed its buffer and crashed the SIP worker; the buffer now fits any ids.
• A joiner whose master died before its key grant arrived read the members' traffic as a wrong password and shut down after about 6 s; it now waits out a master failover (about 30 s) first.
• A joiner admitted by the next master after a failover could wait node_timeout (60 s) for its link to be probed again; a node heard again is now probed at once.
• An unknown key in cluster_options, such as use_controler=1, now fails startup instead of leaving the cluster without the controller.

Also: a joiner still connecting after 60 s is parked, so the admission queue moves on; a node coming online resyncs every capability, not only the first 16; the paths for members without the probe, admission or maintenance capabilities are gone, as every build of wire version 4 has them; and new network-namespace tests cover a 2-2 split, two members cut from each other, the master killed during an admission, and parking. The description has the new measurements: 64 nodes started at once formed in 10 runs of 10, 128 nodes in about four and a half minutes, and replication at 10% loss and with reordering.

[razvancrainea]

Hi, @Lt-Flash !

Thank you for this amazing work! It's ok to have the tm change inside this PR, no worries! Just to double check, currently we can not run with both clusterer and clusterer_controller modules in parallel, right? Perhaps we should make something in the code to prevent loading both of them? Nevertheless, this should be part of a different PR.
Also, just to double check, can you conform that tm + simple clusterer module will continue to work in anycast mode? As long as the two modules implement the same interface, that should not be a problem.

Best regards,
Răzvan

[Lt-Flash]

I have pushed an update: still the same 6 commits, and only the last one, the module itself, changed.

• A master whose own multicast was cut, while it still heard its members, kept the active sharing tag after they had elected another master, so two nodes held it. Its answer about itself no longer holds the members' election, a new master gets its own keepalive watchdog, and a master asked about itself by more than half of the other members within 5 s now steps down until a master hears it again.
• A TCP message that arrived while the broadcast before it was being dispatched could go first; it now waits for it, and one that waits past 500 ms is logged as a warning.
• The channel for short messages between nodes (cl_ctr_send_req and the others) is removed, as nothing used it.
• The README says that controller-managed clusters use plain BIN for now.

New tests: a node whose outgoing multicast is cut, as a member and as the master, and a unit test with three readers, a TCP sender and the broadcast tick running at once. The measurements moved from the description to doc/benchmarks.md.

[Lt-Flash]

Hi @razvancrainea, thank you!

Both modules in one instance: right, they cannot run side by side - both register as clusterer and export its API. This PR already refuses it: clusterer_controller's mod_init counts the modules named clusterer and fails startup with "clusterer_controller.so replaces clusterer.so - load only one of them", and when the script sets the controller's parameters the core refuses them first, as clusterer.so does not know them. The namespace tests check both cases. There is no need for both either: in clusterer_controller a cluster without use_controller=1 runs as with clusterer, so one instance can have controller-managed and static clusters side by side.

tm with clusterer in anycast mode: yes. With clusterer the node id is set at startup (my_node_id, or the node_id of my_node_info) and never changes. tm now reads it per request through the same get_my_id(), so every request carries the same cid as before and a reply's cid is compared with the same id; nothing else in tm's anycast path changed.