Shutdown event sometimes does not trigger when reloading (watchfiles)

[Blue9]

Environment:

macOS 15.6
Python 3.12.11
uvicorn==0.35.0
watchfiles==1.1.0

Per ASGI spec, is it expected that if a server sends a lifespan.startup.complete event (it has completed startup), it should always send a corresponding lifespan.shutdown on shutdown? I'm noticing that this is not the case when running with reloading enabled with watchfiles.

Specifically, the shutdown lifespan event will not run if there are multiple reloads in quick succession (for example if you save a file twice when using WatchFiles). I haven't observed this behavior with StatReload.

Below is a minimal reproducible example:

import asyncio


async def app(scope, receive, send):
    if scope["type"] == "lifespan":
        while True:
            message = await receive()
            if message["type"] == "lifespan.startup":
                print("Server is starting")
                await asyncio.sleep(2)
                await send({"type": "lifespan.startup.complete"})
                print("lifespan.startup.complete")
            elif message["type"] == "lifespan.shutdown":
                print("Server shutting down")
                await asyncio.sleep(2)
                await send({"type": "lifespan.shutdown.complete"})
                print("lifespan.shutdown.complete")
    elif scope["type"] == "http":
        await send({"type": "http.response.start", "status": 200, "headers": [[b"content-type", b"text/plain"]]})
        await send({"type": "http.response.body", "body": b"Hello, World!"})

Save as main.py and run uvicorn --reload main:app (also make sure uvicorn is using watchfiles).
Then, in your code editor, save the file two times quickly (you can increase the asyncio.sleep() duration to make this easier). You should see that the shutdown event is not run when reloading the second time even though lifespan.startup.complete is printed. For example, these are the logs I see when saving twice in quick succession:

INFO:     Will watch for changes in these directories: ['/Users/gautam/uvicorn-test']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [23761] using WatchFiles
INFO:     Started server process [23763]
INFO:     Waiting for application startup.
Server is starting
lifespan.startup.complete
INFO:     Application startup complete.
WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
Server shutting down
lifespan.shutdown.complete
INFO:     Application shutdown complete.
INFO:     Finished server process [23763]
INFO:     Started server process [23778]
INFO:     Waiting for application startup.
Server is starting
WARNING:  WatchFiles detected changes in 'main.py'. Reloading...
lifespan.startup.complete
INFO:     Application startup complete.
INFO:     Started server process [23779]
INFO:     Waiting for application startup.
Server is starting
lifespan.startup.complete
INFO:     Application startup complete.

Server is shutting down is not printed a second time. I believe this is because in Server._serve(), we eagerly return if self.should_exit is already set to True here. If I change this to the following:

await self.startup(sockets=sockets)
if self.should_exit:
    await self.shutdown(sockets=sockets)  # Added
    return
await self.main_loop()
await self.shutdown(sockets=sockets)

The shutdown event always runs when reloading and resources are cleaned up.

Another case is if you press ctrl-C twice in succession, that sets force_exit to true here, and we skip the shutdown event here. I understand if that is expected behavior but that can lead to resource leakage if people rely on the shutdown event to clean up resources, as is suggested by FastAPI in their docs.

[Blue9]

If anyone else is running into these issues, I have a fork here with the changes, adding await self.shutdown(sockets=sockets) and always running the shutdown event, even in the case of a force-exit: https://github.com/Blue9/uvicorn. I can confirm I'm not running into the same issues with this change.

[ramipellumbi]

I am really interested in a solution to this. The current implementation seems like it did not consider the shutdown event being used to clean up resources. Is there a way the uvicorn recommends cleaning up resources prior to shutdown instead?

[Blue9]

Also, a custom signal handler will not work because uvicorn overrides them before re-raising any signals: https://github.com/encode/uvicorn/blob/0.32.0/uvicorn/server.py#L327. Maybe this block of code should only reset the signal handlers if they are self.handle_exit and otherwise leave them as they are. That said, I would prefer to keep all cleanup code in the lifespan event if possible and not have to deal with manual signal handling. Any thoughts here @Kludex?

[sushi4lyfe]

I'm running into this as well, could someone take a look?

[wrabit]

This would be great to have