Len error?

[tmadeytq]

Hi. Anyone run into Len error when using DP with built-in Zookeeper? Seems I can get past this by reducing the size of the patch file. Any way to increase the allowable size? I'm not seeing anything in the doc to assist.
java.io.IOException: Len error 1741295
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:541) ~[delta-server.jar:?]
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332) [delta-server.jar:?]
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522) [delta-server.jar:?]
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154) [delta-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]

[afs]

Hi @tmadeytq -- The size is a zookeeper setting, https://zookeeper.apache.org/doc/r3.7.1/zookeeperAdmin.html , system property: jute.maxbuffer

"When jute.maxbuffer in the client side is greater than the server side, the client wants to write the data exceeds jute.maxbuffer in the server side, the server side will get java.io.IOException: Len error"

It must the same setting on all servers; all the server have to be changed together and
it must not be decreased (without resetting zookeeper to empty).

The all-Zookeeper solution, including storing the patches in Zk, is limited. While convenient, it isn't what ZooKeeper is designed for. Restart will probably get slower over time, and there maybe memory size problems at some point. Storing patches in S3, or MinIO for Azure, gives more scale.

[tmadeytq]

Thanks. Do you know of any limitation in maxbuffer value? We aren't able to use S3, or MinIO in this particular instance unfortunately.

[afs]

There are system claiming to give the S3 API over the filesystem. (And many more provide the other way round file-like access over S3).

Some of the system lists at https://geekflare.com/self-hosted-s3/ claim that - I haven't used any of them.

The file-backed patch server, running with the files on a enterprise filing system or cloud filing system, does provide robustness for the patches. Given the zookeeper restart can slow down as it's database and log grows over time, the fast restart of the file-backed patch server may be a practical solution, especially as replicated zookeeper needs to be administered and maintained.

$job uses a different solution. We use Apache Kafka.