Octopus 0.10.2 / Rails 5.1.5 / Sidekiq workers eventually run out of memory and freeze

[kjvarga]

When upgrading to Octopus 0.10.2 using Rails 5.1.5 we notice that our Sidekiq workers after a day or two will stop processing any jobs. The worker machines, which each run 10 threads, are not even accessible by SSH. They run out of memory. So something in the Octopus upgrade causes them to leak memory over time. Unfortunately we cannot upgrade to Rails 5.2 because of #489 issue with Puma.

This is what the worker node metrics look like when the issue happens: