Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demonstrate the capacity limiter lockup effect with a demo #3033

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

bryce-anderson
Copy link
Contributor

When executing we see the following output in the console:

...
!!!!!!!!!.!..!!!!!!!!!!!!!!!!.!!!!.!!.!.!!!.!!.!..!..!...!....!..!.!!!!!!!!!...!..!!!.!.!!!!..!!!.!.
!!!!!!!!!!!!!!...!!!!.!.!........!...!!!!.!!!!!.......!.!.!.!!.!!....!...!...!!!..!.!.!!!!!!!!!..!.!
!.!!!!...!....!...!.!!.!!....!.!!!!!!!!!!!!.!..!!!!!!!!!!!!!!.!!.!...!....!......!!!!!!!!!!!....!!!!
!!!!!!!!!!!.!...!!!!.!.!!!!..!..!.!.!.!!!!!!!!!!!!..!!!!!!!!!!!!!...!!!!!!!!!!!!!!!!!!!!!!!!.......!!
!.....!..!!!!!!!!!!!...!.!.!..!.!.!!!!.!!!!!!.!!!....!....!.!..!!!!!!!!!!.!!!!!!!!!!!!!!!.!.!!!!!!!!!
!!!!!!......!....!.!!!!!!!!!!...!!!!!..!.!.!..!..!...!!!.!!!!!!!!!!.......!......!....!....!!!!!!!
!!!!!!.!!.!!!!!!!!!!!!!!!.....!!..!.!.!....!......!!.!.!!!..!!!!!!!!!!!!..!.!.!!.!!...!!.!!!!!!!!!!!
..!!!!!!!!!!!!!!!...!!!!!!!!!!!..!!!!.!!!!!...!!!!!!!!!!!!.!!.!!!!!!!!!!!!!!....!.!....!!!!!!!!!!!!!
!!!!..!..!.....!.!!!!!!!!.!.!!...!!!!!..!......!....!!!!.!!!!!!!!!!!!!!!!!!!!!....!.!........!!!.!..
..!..!!!!!!!!!!!!!!!!2024-08-06 10:32:03,251 servicetalk-global-io-executor-1-2 [WARN ] HttpMessageDiscardWatchdogClientFilter - Discovered un-drained HTTP response message body which has been dropped by user code - this is a strong indication of a bug in a user-defined filter. Response payload (message) body must be fully consumed before discarding.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
Consecutive failure threshold reached (200). Terminating.
Terminating.

There is a very strong correlation with lockup and the watchdog filter emitting a message (we don't lockup every time we see a message, but everytime we lockup we see a message right before).

When executing we see the following output in the console:
```
...
!!!!!!!!!.!..!!!!!!!!!!!!!!!!.!!!!.!!.!.!!!.!!.!..!..!...!....!..!.!!!!!!!!!...!..!!!.!.!!!!..!!!.!.
!!!!!!!!!!!!!!...!!!!.!.!........!...!!!!.!!!!!.......!.!.!.!!.!!....!...!...!!!..!.!.!!!!!!!!!..!.!
!.!!!!...!....!...!.!!.!!....!.!!!!!!!!!!!!.!..!!!!!!!!!!!!!!.!!.!...!....!......!!!!!!!!!!!....!!!!
!!!!!!!!!!!.!...!!!!.!.!!!!..!..!.!.!.!!!!!!!!!!!!..!!!!!!!!!!!!!...!!!!!!!!!!!!!!!!!!!!!!!!.......!!
!.....!..!!!!!!!!!!!...!.!.!..!.!.!!!!.!!!!!!.!!!....!....!.!..!!!!!!!!!!.!!!!!!!!!!!!!!!.!.!!!!!!!!!
!!!!!!......!....!.!!!!!!!!!!...!!!!!..!.!.!..!..!...!!!.!!!!!!!!!!.......!......!....!....!!!!!!!
!!!!!!.!!.!!!!!!!!!!!!!!!.....!!..!.!.!....!......!!.!.!!!..!!!!!!!!!!!!..!.!.!!.!!...!!.!!!!!!!!!!!
..!!!!!!!!!!!!!!!...!!!!!!!!!!!..!!!!.!!!!!...!!!!!!!!!!!!.!!.!!!!!!!!!!!!!!....!.!....!!!!!!!!!!!!!
!!!!..!..!.....!.!!!!!!!!.!.!!...!!!!!..!......!....!!!!.!!!!!!!!!!!!!!!!!!!!!....!.!........!!!.!..
..!..!!!!!!!!!!!!!!!!2024-08-06 10:32:03,251 servicetalk-global-io-executor-1-2 [WARN ] HttpMessageDiscardWatchdogClientFilter - Discovered un-drained HTTP response message body which has been dropped by user code - this is a strong indication of a bug in a user-defined filter. Response payload (message) body must be fully consumed before discarding.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
Consecutive failure threshold reached (200). Terminating.
Terminating.
```

There is a very strong correlation with lockup and the watchdog filter emitting a message (we don't lockup every time we see a message, but everytime we lockup we see a message right before).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant