Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Demonstrate the capacity limiter lockup effect with a demo #3033

Closed
wants to merge 14 commits into from
Closed

Demonstrate the capacity limiter lockup effect with a demo #3033

wants to merge 14 commits into from

Commits on Aug 6, 2024

  1. Demonstrate the capacity limiter lockup effect with a demo 

    When executing we see the following output in the console:
```
...
!!!!!!!!!.!..!!!!!!!!!!!!!!!!.!!!!.!!.!.!!!.!!.!..!..!...!....!..!.!!!!!!!!!...!..!!!.!.!!!!..!!!.!.
!!!!!!!!!!!!!!...!!!!.!.!........!...!!!!.!!!!!.......!.!.!.!!.!!....!...!...!!!..!.!.!!!!!!!!!..!.!
!.!!!!...!....!...!.!!.!!....!.!!!!!!!!!!!!.!..!!!!!!!!!!!!!!.!!.!...!....!......!!!!!!!!!!!....!!!!
!!!!!!!!!!!.!...!!!!.!.!!!!..!..!.!.!.!!!!!!!!!!!!..!!!!!!!!!!!!!...!!!!!!!!!!!!!!!!!!!!!!!!.......!!
!.....!..!!!!!!!!!!!...!.!.!..!.!.!!!!.!!!!!!.!!!....!....!.!..!!!!!!!!!!.!!!!!!!!!!!!!!!.!.!!!!!!!!!
!!!!!!......!....!.!!!!!!!!!!...!!!!!..!.!.!..!..!...!!!.!!!!!!!!!!.......!......!....!....!!!!!!!
!!!!!!.!!.!!!!!!!!!!!!!!!.....!!..!.!.!....!......!!.!.!!!..!!!!!!!!!!!!..!.!.!!.!!...!!.!!!!!!!!!!!
..!!!!!!!!!!!!!!!...!!!!!!!!!!!..!!!!.!!!!!...!!!!!!!!!!!!.!!.!!!!!!!!!!!!!!....!.!....!!!!!!!!!!!!!
!!!!..!..!.....!.!!!!!!!!.!.!!...!!!!!..!......!....!!!!.!!!!!!!!!!!!!!!!!!!!!....!.!........!!!.!..
..!..!!!!!!!!!!!!!!!!2024-08-06 10:32:03,251 servicetalk-global-io-executor-1-2 [WARN ] HttpMessageDiscardWatchdogClientFilter - Discovered un-drained HTTP response message body which has been dropped by user code - this is a strong indication of a bug in a user-defined filter. Response payload (message) body must be fully consumed before discarding.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
Consecutive failure threshold reached (200). Terminating.
Terminating.
```

There is a very strong correlation with lockup and the watchdog filter emitting a message (we don't lockup every time we see a message, but everytime we lockup we see a message right before).
    @bryce-anderson
    bryce-anderson committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    5bbf0da View commit details
    Browse the repository at this point in the history

  2. This looks like a case of abandonment, but it may not be the importan… 

    …t one
    @bryce-anderson
    bryce-anderson committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    535ed2b View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Not using BeforeFinallyHttpOperator makes it stable

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    7f9cc71 View commit details
    Browse the repository at this point in the history

  2. See if making response aggregation uncancellable helps

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    91ec87b View commit details
    Browse the repository at this point in the history

  3. Maybe this fixes it?

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    e67103f View commit details
    Browse the repository at this point in the history

  4. Start cleaning up and seeing which changes were necessary

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2ca15bc View commit details
    Browse the repository at this point in the history

  5. remove uncancellable() and it seems stable

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    04ba759 View commit details
    Browse the repository at this point in the history

  6. Add a for test BeforeFinallyHttpOperator

    @bryce-anderson
    bryce-anderson committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    fbad20e View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Merge remote-tracking branch 'origin/main' into bl_anderson/lockup-repro

    @bryce-anderson
    bryce-anderson committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    0ba8976 View commit details
    Browse the repository at this point in the history

  2. Revert some things

    @bryce-anderson
    bryce-anderson committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    d7b9796 View commit details
    Browse the repository at this point in the history

  3. Cleanup output

    @bryce-anderson
    bryce-anderson committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    78072fb View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. More noodling

    @bryce-anderson
    bryce-anderson committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    cda88b5 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Merge branch 'main' into bl_anderson/lockup-repro

    @bryce-anderson
    bryce-anderson committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    6708db4 View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @bryce-anderson
    bryce-anderson committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    fd53b9c View commit details
    Browse the repository at this point in the history