Chunk size configuration & performance results

[oranichu]

Hey again 😇,
Following my last question, is the partial data that is being transferred configurable? can I use bigger or smaller chunks of data for each iteration?
Also, how does it handle big data replication? I heard in one of the early videos (I think it was even in the Scribe videos) that you still need to test the performance for the replication, do you have any statistics of big data replication?

Thanks 💪🏻

[JohnStrunk]

Following my last question, is the partial data that is being transferred configurable? can I use bigger or smaller chunks of data for each iteration?

Not today, but may be possible on a mover-by-mover basis...
The transfer is entirely handled by the specific mover, so if a given mover's engine supports configuring this then it would be possible by exposing that config parameter.
Looking at rsync as an example, it supports the --block-size option that would allow specifying a fixed size. It would be possible to expose this option up through the ReplicationSource's .spec.rsync struct such that a user could configure it per replication pair.

Also, how does it handle big data replication? I heard in one of the early videos (I think it was even in the Scribe videos) that you still need to test the performance for the replication, do you have any statistics of big data replication?

We still haven't done benchmarking on large volumes. I'd be interested in seeing results if you do, though.

[oranichu]

I'm hoping in the near future I'll have time to benchmark on large volumes and I'll share the results.
Thanks 🙏.

[oranichu]

@JohnStrunk just did a manual rsync of around 205 MB and this was the Last Sync Duration: 46.518531386s (from the ReplicationSource)

Going to test a much larger transfer in the near future and update :).