We are using Jersey 3.1.10 with Tomcat 10.1.54. We have compression enabled in Tomcat, and we have an async jaxrs service (using @Suspended AsyncResponse).
We have noticed that one of our instances of our software stack is showing that 2 threads are in a possible livelock situation over a shared GZIPOutputStream / Deflater.
These are the stacktraces of the 2 threads:
"client-response-8" #224 daemon prio=5 os_prio=0
cpu=612621947.84ms elapsed=709472.39s
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflate(Native Method) <-- burning CPU here
at java.util.zip.DeflaterOutputStream.deflate(...)
at java.util.zip.DeflaterOutputStream.write(...)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:148)
- locked <0x000000008ce01a68> (a java.util.zip.GZIPOutputStream)
at org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(...)
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(...)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(...)
at java.io.ByteArrayOutputStream.writeTo(...)
- locked <0x000000008ce02268> (a java.io.ByteArrayOutputStream)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(...)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(...)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(...)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(...)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWrapper.resume(:46)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWithCleanup.resume(:45)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.maybeDeliverBatchResponse(:300)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.lambda$handleSingleRegularResponse$10(:245)
"http-nio-8080-exec-1" #73 daemon prio=5 os_prio=0
cpu=614193943.88ms elapsed=759622.22s
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.zip.Deflater.finished(Deflater.java:442)
- locked <0x000000008ce01a50> (a java.util.zip.Deflater$DeflaterZStreamRef)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:162)
at org.apache.coyote.http11.filters.GzipOutputFilter.end(...)
at org.apache.coyote.http11.Http11OutputBuffer.end(...)
at org.apache.coyote.http11.Http11Processor.prepareResponse(...)
at org.apache.coyote.http11.Http11Processor.action(...)
at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(...)
at org.apache.coyote.AbstractProcessor.dispatch(...)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(...)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(...)
at org.apache.tomcat.util.net.SocketProcessorBase.run(...)
These 2 threads have been running very actively at 80-85% of CPU during the last 8 days, while we barely have any activity in our application (due to a routing issue, most traffic for our application is being directed to other instances). So, from that I assume that these 2 threads are somehow clashing over a shared resource.
I found a similar bug report in the OpenJDK bug tracker:
https://bugs.openjdk.org/browse/JDK-8327687
This bug was closed on the JDK because it is bad usage of the API, the GZipOutputStream is not threadsafe.
According to the Claude 4.6 Thinking model (don't take it as definite proof):
The two lock addresses observed in the dump are separated by exactly 24 bytes on the heap:
0x000000008ce01a50 | Deflater$DeflaterZStreamRef | http-nio-8080-exec-1 holds
0x000000008ce01a68 | java.util.zip.GZIPOutputStream | client-response-8 holds
Objects 24 bytes apart on the heap are consistent with sequential allocation during ZIPOutputStream construction (GZIPOutputStream → Deflater → DeflaterZStreamRef allocated in immediate succession). Both locks belong to the same GZIPOutputStream instance
We are using Jersey 3.1.10 with Tomcat 10.1.54. We have compression enabled in Tomcat, and we have an async jaxrs service (using
@Suspended AsyncResponse).We have noticed that one of our instances of our software stack is showing that 2 threads are in a possible livelock situation over a shared
GZIPOutputStream/Deflater.These are the stacktraces of the 2 threads:
These 2 threads have been running very actively at 80-85% of CPU during the last 8 days, while we barely have any activity in our application (due to a routing issue, most traffic for our application is being directed to other instances). So, from that I assume that these 2 threads are somehow clashing over a shared resource.
I found a similar bug report in the OpenJDK bug tracker:
https://bugs.openjdk.org/browse/JDK-8327687
This bug was closed on the JDK because it is bad usage of the API, the
GZipOutputStreamis not threadsafe.According to the Claude 4.6 Thinking model (don't take it as definite proof):
The two lock addresses observed in the dump are separated by exactly 24 bytes on the heap:
Objects 24 bytes apart on the heap are consistent with sequential allocation during
ZIPOutputStreamconstruction (GZIPOutputStream → Deflater → DeflaterZStreamRefallocated in immediate succession). Both locks belong to the sameGZIPOutputStreaminstance