What network are you wanting to use on the M4?

ssl is built in on devices with native wifi, and also on devices with built-in (WIZnet) Ethernet. There isn't necessarily room to put it on every device, but you can do a custom build (may need to delete other things to make room).

For ESP32SPI, see this thread:
#86 (comment)
and this PR:
adafruit/Adafruit_CircuitPython_ESP32SPI#218

I just entered this on behalf of somebody I helped in the Adafruit Circuitpython forum. I proposed a little change to the code there, that would allow for the import and use on non-SSL-enabled platforms.

Currently, importing the hashlib module works in a similar way.

try:
    try:
        import hashlib
    except ImportError:
        import adafruit_hashlib as hashlib
except ImportError:
    print(
        "WARNING: hashlib module not available and adafruit_hashlib not installed.",
        "Websocket support will not work.",
    )

Maybe this could be applied for ssl too? If there is a use case for adafruit_httpserver on boards that do not have ssl module, in my opinion it should not throw an error just because it is missing a module it wouldn't use either way.