KAFKA-18794: Add check to send one ShareFetchEvent per poll in ShareConsumer. #20794
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Currently in
ShareConsumerImpl, there is a chance we could besending multiple
ShareFetchEventsfor a singlepoll().This could happen when the
ShareFetchBufferis empty in the firstwait and the
pollTimerhas not completed yet, then we would sendmultiple events.
Usually the
ShareFetchBufferwould wait for a time until thepollTimeout, so we would not send multiple events, but the buffer wait
time is also controlled by
applicationEventHandler.maximumTimeToWait()which can return lower values (even 0) in some cases (especially during
startup of heartbeat request manager).
If this happens, we will see multiple events sent, and this could even
lead to multiple fetching from the broker (sort of a fetch and a
pre-fetch) if the response for the previous
ShareFetchEventarrivesbefore the next
ShareFetchEventwas processed.To avoid this, we have a check now which only sends one
ShareFetchEventper poll.This was the reason a couple of tests were flaky in
KafkaShareConsumerTest -
https://issues.apache.org/jira/browse/KAFKA-18794. The PR fixes the
flakiness, now the tests reliably pass locally.
Reviewers: Andrew Schofield [email protected]