Skip to content

Split server’s diff pooling mechanism into a separate module #243 - #253

Open
nngeek195 wants to merge 5 commits into
edgi-govdata-archiving:mainfrom
nngeek195:Split-server
Open

nngeek195 wants to merge 5 commits into
edgi-govdata-archiving:mainfrom
nngeek195:Split-server

Conversation

@nngeek195

@nngeek195 nngeek195 commented Apr 8, 2026 •

Copy link
Copy Markdown

In previously server.py had many things to care, it had to handle web requests and  also at the same time it had to responsible for:

  • Creating and destroying a process pool
  • Counting how many diffs each worker had run
  • Deciding when to recycle workers
  • Recovering from crashed worker processes
  • Deciding whether to shut the whole server down

As the issue #243 explains, the crash recovery logic was tangled inside the HTTP request handler. The pool state was also stored loosely in Tornado's settings dictionary instead of a proper separated object. This made it hard to understand and test independently from the web serving code.
So I extracted all process pool logic into a new class DiffExecutorManager in a new file executor.py.

executor.py now owns:

  • Creating the ProcessPoolExecutor
  • Tracking remaining_diffs to know when to recycle workers
  • Retrying after a BrokenProcessPool crash
  • Graceful vs immediate shutdown of worker processes
  • Raising DiffPoolError when the situation is unrecoverable

server.py is now only responsible for:

  • Receiving HTTP requests
  • Fetching content from URLs
  • Asking the executor manager to run a diff
  • Formatting and returning the response

Because of this change, if the ProcessPoolExecutor crashes, the web server keeps running normally. Only the worker pool gets recycled inside DiffExecutorManager  and the HTTP server and request handling in server.py are completely safe and unaffected.

Close issue(#243)
@Mr0grog

@Mr0grog Mr0grog moved this from Inbox to In Progress in Web Monitoring Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant