Vert.x HttpClient queueing requests on few connections instead of opening new connections

[ppalaga]

Opening a new issue for parts that still do not work in #1853:

Citing @Tomnivore after the Vert.x HttpClient connection pool configuration options were exposed in #1857:

Well from my testing it seems that on the bright side the new configurations work! And some new connections are opened.
On the not so bright side it seems that it wasn't quite the salvation I imagined. I'm in a nasty spot between a rock and a hard place where I have a very limited amount of time to get the answer from the server. Vertx seems to queue a lot of connections on the same threads intstead of opening new connections and the queued requests start to timeout while waiting for their turn. But that's beyond this discussion I think.

@ppalaga Here you go: https://github.com/Tomnivore/minimum-cxf

minimum
Minimum project to reproduce CXF timeouts for SOAP requests.
Both http1-max.size and event-loop-size is put to 100.
Wiremock has a 500ms delay with answering to simulate a slowish response.
When the test in ClientTest.java is run it counts successfull and failure requests as true and false respectively.
It's tested that without the delay 100 requests generate 100 trues.
With the 500ms delay it the result is abut 55 true 45 false.

[ppalaga]

In the first place, sorry @Tomnivore for this issue slipping off my radar, because the reproducer was attached when #1853 was closed already.

I have investigated it a bit an here is what I found:

I had some concerns about whether the WireMock backend is the new bottleneck. Therefore, I implemented an equivalent Vert.x backend in ppalaga/qcxf-2066-vertx-client-bottleneck@f882a41
With that one in place, all requests passed on time.

Then I went researching about WireMock concurrency configuration and found its containerThreads config option. I have increased it to 100 and all requests passed on time - see ppalaga/qcxf-2066-vertx-client-bottleneck@f810b8b

The default of containerThreads is 25, so the theoretical maximum number of 500ms tasks in 2 seconds should be (2sec/0.5sec) * 25 = 100. Why we do not see the corresponding throughput with your original reproducer, is an interesting question. Jetty is perhaps using a part of those threads for other tasks. Anyway, increasing containerThreads improves the throughput and IMO proves that the bottleneck in your reproducer is on the server side.

I'd say that with properly dimensioned backend, Quarkus CXF and Vert.x HTTP client work as designed: The whole test takes 1.260 sec with the Vert.x backend. That's a clear proof that the client is able to leverage "lots" of parallel connections.

[ppalaga]

The way how quarkus.cxf.client."my-client".receive-timeout is counted with the Vert.x client may seem counter-intuitive. The measurement starts even before the client is connected. If you queue many requests, some of those connecting after a delay, then they timeout earlier than one may expect. This is perhaps what you complain about @Tomnivore?

[ppalaga]

Moreover, with URLConnectionHTTPConduit the quarkus.cxf.client."my-client".receive-timeout value is passed to java.net.URLConnection.setReadTimeout(int) which is something fully different.
It is quite far from the description in org.apache.cxf.transports.http.configuration.HTTPClientPolicy.getReceiveTimeout():

Specifies the amount of time, in milliseconds, that the consumer will wait for a response before it times out. 0 is infinite.

Let me check how other HttpConduit implementations interpret quarkus.cxf.client."my-client".receive-timeout.

[ppalaga]

HttpClientHTTPConduit passes the receive-timeout value to java.net.http.HttpRequest.Builder.timeout(Duration) which perfectly matches the JavaDoc in org.apache.cxf.transports.http.configuration.HTTPClientPolicy.getReceiveTimeout().

It seems that java.net.http.HttpClient starts counting once it connects to the server. That would make sense, because there is quarkus.cxf.client."client-name".connection-timeout for the time before connecting.

So perhaps we should move the start of receive timeout measurement in VertxHttpClientHTTPConduit to the point when the client connects to the server too.