[WIP] Copy-on-Write forking (logic copied from Puma) #1160
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reason: The `script/delayed_job` process (described above) uses the [Daemons](https://github.com/thuehlinger/daemons) gem which has several undesirable behaviors when running inside a container (Docker, etc.): - The parent process spawns a background child process then exits immediately, which causes the container to shutdown. - The worker processes are detached from the parent, which prevents logging to `STDOUT`. The `--fork` option solves this by running workers in a foreground process tree. When using `--fork` the `daemons` gem is not required. The command script requires changing the following line to enable this: # always daemonizes, never forks Delayed::Command.new(ARGV).daemonize must now be: # daemonize by default unless --fork specified Delayed::Command.new(ARGV).launch In addition, there are several quality of life improvements added to the Command script: - Command: Add --pools arg as an alias for --pool, and allow pipe-delimited pool definitions - Command: Add --num-worker arg as an alias for -n / --number-of-workers (and deprecate --number-of-workers) - Command: Pool parsing code has been extracted to PoolParser class - Command: Add -v / --verbose switch which sets quiet=false on workers - Command: Add -x switch as alias for --exit-on-complete - Command: Add STDERR warning num-workers less than 1 - Command: Add STDERR warning if num-workers and pools both specified (pools overrides num-workers as per existing behavior) - Command: Add STDERR warning if queues and pools both specified (pools overrides num-workers as per existing behavior) The Rake script has also been enhanced: - Rake: Uses Forking launcher under the hood - Rake: Add support for NUM_WORKERS and POOLS args
- Add validation of min vs. max priority
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR further improves #1138 by adding copy-on-write forking which forks from the first worker child to the other children. After a configurable timeout (e.g. 1 hour), it will re-fork its children from the first child.
The reason is to reduce memory usage and our AWS bill at TableCheck :)
Much of the code is copied from Puma 5, with proper attribution. (Puma is MIT licensed.)
TODOS:
restart,stop, etc. script commands.