Executor queue grows unlimited and cause OoM

[tomasz-sadura]

ExecutorCompletionService in DynamoDBConsumer have tasks submitted, but the results are never taken from the executor completion queue. This may cause OutOfMemory errors for large tables.

[jseed]

+1 For this issue.
Attempting to copy larger tables causes the import export tool to fail, throwing java.lang.OutOfMemoryError: GC overhead limit exceeded

[viyatgandhi]

@jseed can you tell me how big of a table can cause this problem ? I am thinking to use this in prod for some replication issue, although it worked fine in our test env on small table.

[amcp]

To limit the size of your executor queue and avoid OutOfMemory errors, you can take the following steps:

1. Increase the memory you allocate to the JVM
2. In case you have already maxed out JVM allocation, try doing the table copy in multiple sections. Tune the number of sections until the memory footprint of each section fits within your JVM allocation.
3. You can copy sections serially on the same host, or in parallel on multiple hosts.

[amcp]

I agree that this library should use numWriteThreads somehow to limit the queue length.