Parrallel Producer per partition

[suyashtava]

Issue. #395

This PR aims to provide parallel Producer per partition, so that even if one partition is slow(hosted on slow Brtoker) other producers can easily produce data on other partitions.

[suyashtava]

Previous Discussion: #375

[mhratson]

@suyashtava thanks for the PR, have your observed the memory consumption increase with this approach?
Do you have some data to share?

[suyashtava]

@suyashtava thanks for the PR, have you observed the memory consumption increase with this approach? Do you have some data to share?

@mhratson There is a small increase in memory. I don't have any comparison data ready at the moment since we changed this months ago in production.

GIve me a day I will reproduce the comparison data on the Weekend.

[suyashtava]

@mhratson
KMF is running on a Kafka Cluster with total of 12 Brokers, and each Broker is hosting 3 partitions of KMF.

With the existing opensource KMF: It is taking avg 73 MB of Memory, wherease with Parallel Producer(PR): It is taking avg 85 MB of memory.

CPU Usage almost remains untouched.

FYI: We are using java.lang.Memory.HeapMemoryUsage.usedBytes for memory calculation.

[mhratson]

Have you considered using using async send as alternative? That'd require changing metrics collection a bit rather than than constructing per-partition producer