Add feature to enable connection diversity to a single upstream

[howardjohn]

Title: Add feature to enable connection diversity to a single upstream

Description:
We would like the ability to tune the HTTP connection pool logic to create more connections to certain upstreams.

Typically, each upstream would have only 1-2 connections associated with them. This can be problematic when the endpoint represents multiple true backends behind it.

For example, say I am hitting storage.googleapis.com. Using a single connection for all my requests will ultimately be slower. This is explicitly recommended/used by Google APIs in a few places:

Developers on Google’s cloud platform will now also be able to upload their files faster thanks to Gsutil 3.4, which now uploads large objects in parallel. This update automatically uploads larger files over multiple connections to increase TCP throughput [1]

As we found out, if those file names are sequential then you could end up in a situation where multiple connections get squashed down to a single upload thread (thus hurting performance) [2]

Another case is when we are hitting a non-HTTP load balancer. istio/istio#46959 is one such example.

Another example is Istio multi-network. In this model, we have an EDS config looking like this:

priority0:
  - local-pod-0
  - local-pod-1
  - local-pod-2
priority1:
  - single-remote-gateway-ip
    weight: 7 # represents workloads behind this one IP

In this case, we want to introduce connection diversity only to single-remote-gateway-ip. Without this, we will likely open up only 1-2 connections and hit only 1-2 backends on the remote side, resulting in uneven load. If we were to open up more connections, we have a much more probabilistic chance of having a more even load.

[yanavlasov]

@howardjohn is this specific to H/2 or H/3 protocols? I.e. you want Envoy to not shared multiplexed connections to some upstreams?

[howardjohn]

It is not specific to any version, I would want it for all protocol versions. HTTP/1.1 has the same issue unless you have enough concurrent load to trigger multiple connections

[yanavlasov]

Is it known how many actual hosts are behind a VIP? If not, how can Envoy decide whether to open a new connection or reuse an existing one?

[howardjohn]

In some cases yes, in others no.

In the event we do, we probably want a factor based on the count. It's not always 1:1 IMO - if I have 2 hosts and open 2 connections, I have a 50% chance to put both on the same host. I may want a 2x factor and have 4 connections for example.

If we don't (for example, hitting googleapis.com) likely a fixed count is sufficient?