Add in-line deduplication for TCP payload

[mark-stopka]

Hello, I believe it would be very beneficial for OpenNOP to support deduplication of TCP payload. I know about https://github.com/centeropenmiddleware/solowan which implements de-duplication and about this feature branch (https://github.com/yaplej/opennop/tree/feature/deduplication) implementing deduplication from @yaplej.

I was wondering if there is a reason why deduplication from SoloWAN was not merged into OpenNOP master and why new deduplication feature is being implemented. I started to work on porting SoloWAN deduplication onto master (https://github.com/branch-os/opennop/tree/merge_with_solowan) before finding out about @yaplej feature branch, and I wonder if I should continue in my efforts or if I should rather contribute towards the new feature branch.

[yaplej]

Yes, I agree that it would be very beneficial. The SoloWAN project jumped slightly ahead of OpenNOP in this feature to meet their needs and re-released the modification they made. They never requested the changes be merged back into OpenNOP and I have not reviewed all of the SoloWAN code.

Currently deduplication is the only feature I am now working on because I feel that the rest of OpenNOP is to a point where other underlying features are now present to implement it to its full potential.

Some of the capabilities of the deduplication engine I started working on are:

1. Block Based (128 byte blocks reusable in any payload).
2. Uses SHA512 as hashing method.
3. Persistent deduplication into Berkeley DB.
4. Multi-node aware. (mapping blocks to remote neighbors).
5. Variable Block Sizes (future capability with combining Merkle Tree and multiple levels of blocks).