Questions about network configuration and weird behavior shown in prometheus

[danielporto]

Hello there!

Is the TPS curve supposed to stabilize? Are the clients openloop? How many clients can saturate a network with 4 nodes?

A bit of context:
I've started generating my network (ibft 4 nodes) with quorum wizard using baremetal approach. Then I update the script to work in a real distributed scenario. each process at its own server. Quorum and Tessera are colocated.

To test the network I used quorum-profiling. I have it configured as a single distinguished node, issuing requests (default public contract) to all 4 quorum nodes in a total of 4 threads. I noticed a strong queuing effect and a very poor performance.

Long story short, some tweaks helped a lot:

• changed the quorum logging level from 5 (quorum-wizard default ) to 2 increased TPS a lot
• changed the mine threads from 1 (quorum-wizard default) to 10 also improved the performance substantially
• changed the ibft blockperiod from 5 to 1 also improved the performance a lot

with these 3 changes alone, the performance went from a couple hundreds to >1k TPS
Nevertheless, there is a strange behavior I observe in prometheus that remains: looking a the TPS behavior over time (tried with 5, 10, 20min) there is a initial spike, in the plot, then a "sawtooh" shape appears. The TPS never stabilizes which is odd specially because there is a single host with 4 threads issuing requests. Which I didnt expect to saturate the system.
I attached the figure. The experiments are split in time. thus some space between the curves. In the first plot, there is the same spike (sawtooth) then long tail shape even if the clients continue to issue requests it queues up ... please also note that from the other figures the block generation stops and also the number of transactions. That happen with a single client issuing requests.

I was expecting it to stabilizes at some point.

Therefore I would like your input on this behavior and in what configuration I shall make to achieve a stable (correct?) behavior.
There are other parameters (geth) I didn't change: namely accountsslots, globalslots, accountqueue, globalqueue. What would you recommend?
Thanks a lot