Intelligent client connection load balancing

[rymarm]

One drillbit may end up with significantly more client connections than others.

Here is an example of such a case:

The chart shows that the highest number of connections per node is 21, while the lowest is only 3 – a difference of 7 times.

Even though query execution considers the load on all Drillbits and distributes execution accordingly, overutilizing a single Drillbit for client connections can lead to issues such as heap memory exhaustion and excessive network bandwidth usage when returning query results to the client.

The current client implementation retrieves a list of active Drillbits from Zookeeper, randomly shuffles them:

java.util.Collections.shuffle(endpoints);

and attempts to connect to the first N nodes until a connection is successful. This approach was introduced as a simple way to prevent all clients from connecting to the same Drillbit: https://issues.apache.org/jira/browse/DRILL-2512. However, it does not fully solve the problem, as one node may still end up with significantly more connections than others.

Solution
Every drillbit stores the his current number of client connections in Zookeeper. After retrieving a list of active drillbits, the client calculates the average number of connections and reorders the list based on whether a drillbit's connection count is above or below the average before attempting a connection.

The key questions I see here are:

• Is it appropriate to store information about the current number of user connections in Zookeeper?
• Should client connection load balancing be enabled be configurable and enabled by default?
• Is it beneficial to distribute all connections evenly across the entire Drill cluster?

[rymarm]

@jnturton @cgivre Hi! What do you think about the idea of balancing client connections more evenly across the Drill cluster? Do you think it could improve overall performance, or could it introduce other challenges? I'd love to hear your thoughts and opinions on how this might play out.