Avoid synchronization on GetQueues

[klockla]

The purpose of this PR is to avoid the synchronization on the internal queue map.

The synchronized(getQueues()) call causes contention during long queue traversals which are very likely to occur during calls to countURLs & ListURLs.

This change is based on the ConcurrentOrdereredMap class which uses StampLock to provide fine
fine-grained locking over the concurrent map.

(alternative in ConcurrentLinkedHashMap based on ReentrantLock, need to compare under stress tests)

Signed-off-by: Laurent Klock [email protected]

[jnioche]

Thanks @klockla, sounds good
Would be great to have a test or benchmark to compare the existing approach vs ConcurrentOrdereredMap vs ConcurrentLinkedHashMap indeed.
Given that the same approach is used in key parts of StormCrawler it would be great to have a better approach.
Is the code for ConcurrentOrdereredMap and ConcurrentLinkedHashMap from another project or did you write that? We would need a strong set of unit tests to make sure they are bullet proof

[klockla]

Hello @jnioche
I tried some other implementations with different combinations of data structures and locking mechanisms.
So far, the best candidate according to my tests is ConcurrentStripedOrderedMap (yes, I'll change this name later :) )
See some benchmarks in the attached Excel file

ConcurentOrderMapTests.xlsx

I'm going to drop the others..but your feedback is welcome

[jnioche]

Thanks. What's the difference between Bench, ReadHeavy and WriteHeavy?
Is Synchronized LinkedHashMap what we currently have and isn't it performing better on Bench and WriteHeavy?
Is this code taken from somwhere or did you write it from scratch?

[klockla]

The code is written from scratch. You can see the different performance tests in ConcurrentMapStressTest.java
The purpose of these tests are to test the performance under heavy load with many threads doing many ops on the maps.

Basically, in bench every thread are just doing put, gets and a remove every 2 loop cycle.
ReadHeavy is mostly doing gets and a put every 1000 cycle (on a pre initialized map)
WriteHeavy is the opposite: a put every cycle, a get every 100 cycle and a remove every 500.
Yes, the synchronized LinkedHashMap is performing a bit better on bench and WriteHeavy but the whole idea of the thing
is to have a data structure that avoids a global synchronization while we're iterating over its elements.