9.2.0 (esp-idf 5.3.x) and later on ESP32-C3 boards: DEBUG builds crash

[dhalbert]

DEBUG builds stopped working with ESP32-C3 boards with esp-idf 5.3.x. It causes a crash very early in startup, before main, which appears to be due to serial startup problems. See below for details.

9.1.4 works; 9.2.0-beta.0 (which starts with ESP-IDF 5.3.1) does not.

ports/espressif/esp-idf-config/sdkconfig-debug.defaults was changed between these versions to include:

CONFIG_ESP_CONSOLE_SECONDARY_NONE=y
# CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG is not set

Taking these away, singly or together, or changing them to the opposite value, causes link errors like:

/home/halbert/.espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/13.2.0/../../../../riscv32-esp-elf/bin/ld: build-adafruit_qtpy_esp32c3/esp-idf/esp-idf/esp_vfs_console/libesp_vfs_console.a(vfs_console.c.obj): in function `esp_vfs_console_register':
/home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_vfs_console/vfs_console.c:197:(.text.esp_vfs_console_register+0x10): undefined reference to `esp_vfs_usb_serial_jtag_get_vfs'

Here is a typical crash (as printed on TX/RX on QT Py ESP32-C3)

I (551) heap_init: At 3FCDC710 len 00002950 (10 KiB): Retention RAM
I (558) heap_init: At 50001204 len 00000DE4 (3 KiB): RTCRAM
Guru Meditation Error: Core  0 panic'ed (Instruction access fault). Exception was unhandled.

Core  0 register dump:
MEPC    : 0x00000000  RA      : 0x00000000  SP      : 0x3fcde220  GP      : 0x3fc9b000  
TP      : 0x00000000  T0      : 0x4005890e  T1      : 0x40391fd2  T2      : 0xffffffff  
S0/FP   : 0x3fcac75c  S1      : 0x3fc9c9a4  A0      : 0x00000001  A1      : 0x00000000  
A2      : 0x0000000c  A3      : 0x4005891a  A4      : 0x00002000  A5      : 0xffffe000  
A6      : 0x00000800  A7      : 0x0000002b  S2      : 0x00000800  S3      : 0x00000080  
S4      : 0x3fcde434  S5      : 0x0004b7f0  S6      : 0x3c160000  S7      : 0xffff0000  
S8      : 0x00060020  S9      : 0x00010020  S10     : 0x00000007  S11     : 0x3fcde3d0  
T3      : 0x00000000  T4      : 0x4209a07e  T5      : 0x4209a072  T6      : 0x4209a066  
MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x00000001  MTVAL   : 0x00000000  
MHARTID : 0x00000000  

Stack memory:
3fcde220: 0x00000000 0x00000000 0x00000000 0x00000001 0x00000000 0x3c19b590 0x3c17114c 0x3c163000
3fcde240: 0x3c17114c 0x3fc9c9a4 0x3fcac75c 0x42143504 0x3c17114c 0x3fc9d000 0x3fc9c9a4 0x42094988
3fcde260: 0x3c160000 0x0004b7f0 0x3fcde434 0x0000003e 0x00000000 0x3c1ab810 0x3c1ab7d0 0x420949a8
3fcde280: 0x00000000 0x3c1ab810 0x3c1ab7d0 0x420913d6 0x00000002 0x3fcde2e4 0x00000004 0x421507a4
3fcde2a0: 0x42000000 0x3c1ab000 0x3c1ab000 0x42091428 0x42000000 0x00000001 0x00000000 0x40383724
3fcde2c0: 0x400391e6 0x00000000 0x00000006 0x40039258 0x0000e000 0x3fcd6000 0x00000000 0x00000006
3fcde2e0: 0x03700000 0x3ff10101 0x00000000 0x7fffffff 0x3c160000 0x0004b7f0 0x3fcde434 0x40383482
3fcde300: 0x42000000 0x3c1ab80f 0x4215a9a7 0x403cf812 0x91d35daa 0x000001d0 0x0015a988 0x00010000
3fcde320: 0x00000002 0x000003e8 0x00000000 0x00160000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde340: 0x00000000 0x00000000 0x00000000 0x3fcdf000 0x000054c0 0x3ff1c14c 0x0000ffff 0xffffffff
3fcde360: 0x00000000 0x3fcde4b0 0x00000000 0x403cfc58 0x400481b0 0x00010000 0x00200000 0x00010000
3fcde380: 0x2f0207e9 0x40383482 0x000000ee 0x00000005 0x00ffff00 0x01000000 0x3c160020 0x0004b7f0
3fcde3a0: 0x3fc9a800 0x00004800 0x42000020 0x0015a988 0x3fc9f000 0x00000140 0x40380000 0x0001a61c
3fcde3c0: 0x50000000 0x00000200 0x50000200 0x00001004 0x00000000 0x00000000 0x00000000 0x00000000
3fcde3e0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde400: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00010020 0x0005b818
3fcde420: 0x00060020 0x001ba9b0 0x001baaf8 0x001d511c 0x001d5324 0x00000000 0x00000000 0x00000000
3fcde440: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x001c6350 0x91d35daa
3fcde460: 0x59e1f460 0xf4861c01 0xab59cd6d 0x9799a23a 0x1c0eef47 0x7ff14b5f 0x3344810b 0x00000000
3fcde480: 0x000054c0 0x3ff1c14c 0x0000ffff 0x3fce0000 0x00000000 0x000000bb 0x000054d0 0x403cc79e
3fcde4a0: 0x5b548c35 0x8ef8dede 0x00000000 0x00000000 0x0000e000 0x00002000 0x00000000 0x00000000
3fcde4c0: 0x00000000 0x00000000 0x00010000 0x00200000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde4e0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde500: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde520: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde540: 0x00000000 0x00000000 0x00000001 0x00000000 0x00000000 0x000000bb 0x000054d0 0x4004a296
3fcde560: 0x586337bd 0x3c0d11ec 0x3fcde668 0x00000000 0xffffff64 0x00000000 0x00010000 0x2f0203e9
3fcde580: 0x403cc71a 0x403ce710 0x00002dcc 0x000000ee 0x03030005 0x0000c700 0x01000000 0x3c0054d0
3fcde5a0: 0x7533885e 0x89f520a1 0x8efe0712 0x697c5fc6 0x40999c08 0xb31abd13 0x9a0128e2 0xbe40ba9d
3fcde5c0: 0xc09ffe94 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
3fcde5e0: 0x00000000 0x3fcde700 0x00000101 0x00000002 0x89f520a1 0x8efe0712 0x697c5fc6 0x40999c08
3fcde600: 0xb31abd13 0x9a0128e2 0xbe40ba9d 0xc09ffe94 0x00000000 0x00000000 0x00000000 0x00000000



ELF file SHA256: 70c108f74

CPU halted.

tools/decode_backtrace.py produces the output below. Notice the esp_vfs_console addresses.

 ? ? 0x3fc9b000: __global_pointer$ at ??:?
? 0x40391fd2: esp_ptr_executable at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_hw_support/include/esp_memory_utils.h:232
 (inlined by) vPortTLSPointersDelCb at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/freertos/FreeRTOS-Kernel/portable/riscv/port.c:733
 (inlined by) vPortTCBPreDeleteHook at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/freertos/FreeRTOS-Kernel/portable/riscv/port.c:764
? 0x3fc9c9a4: __global_pointer$ at /builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/sys/reent.h:771
? ? ? 0x3c160000: _flash_rodata_dummy_start at ??:?
? ? 0x4209a07e: console_tcflush at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_vfs_console/vfs_console.c:159
0x4209a072: console_tcdrain at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_vfs_console/vfs_console.c:154
0x4209a066: console_tcgetattr at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_vfs_console/vfs_console.c:149
? 0x40380001: _vector_table at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/riscv/vectors_intc.S:54
? ? ? ? 0x3c19b590: ?? at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_vfs_console/vfs_console.c:163
? 0x3fc9c9a4: __global_pointer$ at /builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/sys/reent.h:771
0x42143504: __swsetup_r at /builds/idf/crosstool-NG/.build/riscv32-esp-elf/src/newlib/newlib/libc/stdio/wsetup.c:73
0x3fc9d000: __global_pointer$ at ??:?
0x3fc9c9a4: __global_pointer$ at /builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/sys/reent.h:771
0x42094988: esp_newlib_init_global_stdio at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/newlib/newlib_init.c:201
? 0x3c160000: _flash_rodata_dummy_start at ??:?
0x3c1ab7d0: _binary_x509_crt_bundle_end at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/newlib/newlib_init.c:206
0x420949a8: __esp_system_init_fn_init_newlib_stdio at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/newlib/newlib_init.c:214
? 0x3c1ab7d0: _binary_x509_crt_bundle_end at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/newlib/newlib_init.c:206
0x420913d6: do_system_init_fn at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/startup.c:128
0x421507a4: esp_mprot_set_prot at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_hw_support/port/esp32c3/esp_memprot.c:716 (discriminator 1)
? 0x3c1ab000: _binary_x509_crt_bundle_end at ??:?
0x3c1ab000: _binary_x509_crt_bundle_end at ??:?
0x42091428: do_global_ctors at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/startup.c:93
 (inlined by) start_cpu0_default at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/startup.c:201
0x40383724: call_start_cpu0 at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/port/cpu_start.c:843
? ? 0x3c160000: _flash_rodata_dummy_start at ??:?
0x40383482: call_start_cpu0 at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/port/cpu_start.c:393
? 0x3c1ab80f: _binary_x509_crt_bundle_end at ??:?
? ? ? ? 0x40383482: call_start_cpu0 at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_system/port/cpu_start.c:393
0x3c160020: ?? at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_app_format/esp_app_desc.c:19
? 0x3fc9a800: ?? at /builds/idf/crosstool-NG/.build/riscv32-esp-elf/src/newlib/newlib/libc/stdlib/environ.c:36
0x42000020: nlr_push_tail at /home/halbert/repos/circuitpython/ports/espressif/../../py/nlr.c:43
0x3fc9f000: __global_pointer$ at ??:?
0x40380000: _vector_table at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/riscv/vectors_intc.S:54
? 0x50000000: esp_default_wake_deep_sleep at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_hw_support/sleep_modes.c:404
0x50000200: ?? at /home/halbert/repos/circuitpython/ports/espressif/esp-idf/components/esp_hw_support/sleep_modes.c:274
? ? ? ? ? ? ? ? ? ? ? ? ? 0x3c0d11ec: _flash_rodata_dummy_start at ??:?
? 0x3c0054d0: _flash_rodata_dummy_start at ??:?