Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Increase http client's max chunk size #10071

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Gerrrr
Copy link
Contributor

@Gerrrr Gerrrr commented Sep 21, 2023

Description

What behavior do you want to change, why, how does your patch achieve the changes?

Testing done

Describe the testing strategy. Unit and integration tests are expected for any behavior changes.

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")
  • Do these changes have compatibility implications for rollback? If so, ensure that the ksql command version is bumped.

@cla-assistant
Copy link

cla-assistant bot commented Sep 21, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ lucasbru
❌ Gerrrr
You have signed the CLA already but the status is still pending? Let us recheck it.

Comment on lines +118 to +121
new HttpClientOptions().setMaxChunkSize(CHUNK_SIZE),
Optional.of(new HttpClientOptions()
.setProtocolVersion(HttpVersion.HTTP_2)
.setMaxChunkSize(CHUNK_SIZE)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure you need to increase the chunk size on the client and not on the server?
The post this PR is based on mentions the server. Are these options propagated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are client options, not server options. We need to make the change to the server I believe. Not sure what happens if it's out of agreement with the client.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Google Group post described an issue on the server side. Check the classes at bottom of the stack trace:

java.lang.IndexOutOfBoundsException: writerIndex: 10614 (expected: readerIndex(0) <= writerIndex <= capacity(8192))
        at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:118)
        at io.netty.buffer.CompositeByteBuf.writerIndex(CompositeByteBuf.java:1686)
        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:851)
        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:228)
        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:51)
        at io.vertx.core.streams.impl.PumpImpl.lambda$new$1(PumpImpl.java:64)
        at io.vertx.core.http.impl.HttpServerRequestImpl.handleData(HttpServerRequestImpl.java:373)
        at io.vertx.core.http.impl.ServerConnection.handleChunk(ServerConnection.java:293)
        at io.vertx.core.http.impl.ServerConnection.processMessage(ServerConnection.java:435)
        at io.vertx.core.http.impl.ServerConnection.handleMessage(ServerConnection.java:131)**
        at io.vertx.core.http.impl.HttpServerImpl$ServerHandler.doMessageReceived(HttpServerImpl.java:678)
        at io.vertx.core.http.impl.HttpServerImpl$ServerHandler.doMessageReceived(HttpServerImpl.java:573)

while the stack trace we observed happens on the client side. Check the classes at bottom of the stack trace:

java.lang.IndexOutOfBoundsException: writerIndex(8192) + minWritableBytes(7569) exceeds maxCapacity(8192): UnpooledSlicedByteBuf(ridx: 0, widx: 8192, cap: 8192/8192, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 16384, widx: 16384, cap: 16413))
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:294)
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103)
	at io.vertx.core.buffer.impl.BufferImpl.appendBuffer(BufferImpl.java:268)
	at io.vertx.core.parsetools.impl.RecordParserImpl.handle(RecordParserImpl.java:283)
	at io.vertx.core.parsetools.impl.RecordParserImpl.handle(RecordParserImpl.java:27)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51)
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:239)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:443)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:678)

I thought that these are 2 similar issues, but happening in different places; hence, changed configuration on the client side.

I agree with @AlanConfluent that we should also change the chunk size on the server side to make client and server config match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants