ESP32-S* Safe Mode when using HTTPS / TLS TCP server #9937
Comments
|
Possibly related #9003 |
|
Tested ESP32-S2 (Feather TFT) with the fix from "espressif: Allow mbedtls data in psram if available" #9867. The memory fix allows HTTPS server to run on ESP32-S2, but S2 is prone to intermittent Safe Mode like S3. |
anecdata
changed the title
|
This still sounds like there is some stray pointer or use-after-free or something going on. The ability to use PSRAM may make the bug less likely. |
|
In continued testing, I've still seen no safe modes with blocking |
|
Generated an ELF map for the Likely same bug as #9003. |
|
This could be a memory exhaustion issue. In Likewise, |
|
With some debug tracing added and using the test server from #9428, I'm seeing |
|
#9173 is the only one that does not use httpserver / accept. |
|
Occurred to me that the observation I made about blocking versus non-blocking may just be a matter of time. Non-blocking hammers on the accept call, but blocking may eventually crash too though in orders of magnitude more time. |
|
As observed earlier, each time The heuristic used in |
|
After a fruitful chat with @dhalbert getting some traction on debug. Found multi-heap integrity testing code in IDF: The PC is trash, but the stack pointer is consistent for the 3 failures I've produced so far. To me this suggests that |
|
Was able to catch a failure with gdb over JTAG running. In the case I caught, crash was due to a double fault on thread 4: In this instance the thread is crashing on a memory access error due to a corrupted stack pointer: Working backwards, the Sleuthing the registers that do point to valid memory, The crashes I've observed do vary, so I'm running again hoping for a crash with more context. |
|
Breaking into a running system, thread 4 is the CircuitPython thread. Thread 9 (the tiny_usb thread) shows a valid stack, whereas for the crash case it didn't. |
CircuitPython version
Code/REPL
Various comments and example code using
adafruit_httpserverlibrary and manual TLS TCP socket server at: adafruit/Adafruit_CircuitPython_HTTPServer#88Using latest HTTP Server library from that PR. Safe mode occurs with 9.2.1, or with
DEBUGbuild from recent tip of main.It is not necessary to have any client attempt to connect.
Behavior
Intermittent safe mode when running HTTPS servers, can be several minutes or a couple hours.
DEBUGoutput shows crashes on Core 0 or on Core 1, with varying error messages. Not a lot of consistency.DEBUGbacktraces are either corrupted or yield no useful output (decode_backtrace.pyjust prints a number of blank lines).Changing to a blocking
accept()call seems to fix the issue, or at least I have not observed any Safe Mode occurrences when using blockingaccept().Tested predominantly on adafruit_feather_esp32s3_tft and waveshare_esp32_s3_eth (using wifi). @michalpokusa observed the issue on a Memento board.
Example adafruit_feather_esp32s3_tft debug output
Example waveshare_esp32_s3_eth debug output
Description
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: