mg_error peer RST during firmware upload

[rossack]

I'm using the sample code from mongoose/tutorials/http/device-dashboard with the following config:
#define MG_ARCH MG_ARCH_PICOSDK
#define MG_ENABLE_TCPIP 1
#define MG_ENABLE_DRIVER_PICO_W 1
#define MG_ENABLE_PACKED_FS 1
#define MG_OTA MG_OTA_PICOSDK
#define MG_IO_SIZE 4096

and mg_mgr_poll(&mgr, 1);

During the file upload I get a random lost(?) HTTP response which the client never receives. Example debug messages when the error occurs:
949a 2 net.c:192:handle_firmware_uplo File level_monitor.bin, offset 49152, len 2048
949f 3 net.c:248:fn 2 POST /api/firmware/upload -> 200
ab41 3 mongoose.c:5408:rx_arp ARP: tell 192.168.0.16 we're 28:cd:c1:05:b0:0a
dfa7 1 mongoose.c:685:mg_error 2 0 peer RST
dfab 2 net.c:192:handle_firmware_uplo File level_monitor.bin, offset 49152, len 2048
dfb1 3 net.c:248:fn 3 POST /api/firmware/upload -> 200

The problem is not in the pico specific code (where interrupts get disabled) as the problem still occurs when I dummy out the mg_ota_flash_xxx() calls;

Very grateful if one of the gurus can tell me what Im missing?

[scaprile]

What you're missing is a Wireshark capture and all the data to be able to match that call to mg_error() to who is doing it, in order to determine the reason for it.
It would be great if you could use an in-circuit debugger and have a call trace, too.
I guess something goes on between ab41 and dfa7 ? Maybe there's a clue in there ? Otherwise you need to either determine the reason of the RST from the traffic or the call stack

[rossack]

When the error occurs, I can see the client POST request with Wireshark:
POST /api/firmware/upload?offset=30720&total=355308&name=level_monitor.bin Content-Length: 2048
TCP Flags: [PSH, ACK]
and the server responds with:
[TCP ACK] Seq=1651 Ack=39005
but the HTTP 200 response which should follow is not sent, even though the MG_DEBUG message shows:
589d 2 net.c:192:handle_firmware_uplo File level_monitor.bin, offset 22528, len 2048
58a2 3 net.c:244:fn 2 POST /api/firmware/upload -> 200
The client starts retransmitting:
[TCP Retransmission] Seq=39005 Ack=1651 Len=588
and eventually resets the connection:
Frame 536: [RST, ACK]

The server is still responding to other HTTP requests:
338e6 3 net.c:244:fn 3 GET /api/stats/get -> 200
but firmware upload is broken.
I'm using PicoSDK 2.2.0. I can only run Wireshark at the client side as the Pico is connected via WiFi.
The failure point seems random, if I wait long enough the upload completes (mostly with error)
I have tried reducing chunk size, different .bin files, standing on one foot,,,
Clutching at straws: Buffer sizes? Stack or heap exhaustion? Something between client and server?

[rossack]

Forgot to mention, I'm running the server under VSCode Cortex Debug build. So I can set breakpoints/watchpoints but I don't know the Mongoose TCP/IP stack well enough to dig this deep

[scaprile]

Thanks for your trying, here is the Rafiki situation: I don't need what you see, I need what you don't see. You are asking a question, if you knew the answer you wouldn't be asking a question, so the answer is not in what you know, it is in what you don't know.
Mongoose does not handle retransmissions; except for the obvious cases.
There were a lot of unknowns to me here, I was just guessing from what you wrote, now you have shined some light; if you're using Wi-Fi and there's an important frame loss, then chances are the TCP connection breaks. It can be something we know it happens, it can be something we didn't knew it happened. Regular transfers are small, OTAs are large, loss probability for a single segment is the same, expected value for a long chain of segments rises.
To see what happens, I need to see what is going on. Then, we can decide what to do with that.

[rossack]

Thanks for your comments. Wireshark is only telling me the HTTP 200 was not received, which doesn't mean the server didn't send it. Both client and server are on same subnet and I tried connecting the PicoW to a different Wifi AP with same result so, I don't think the response frame is being sent by the pico driver.
OTA's are large, but so are Wizard based OTA's which uses a single Post(?) for the upload, which works.
However, if I use a Wizard generated app I see lots of (error) frames during the upload like:
[TCP Retransmission]
[TCP Dup ACK 212500#1]
[TCP Window Full]
[TCP Dup ACK 212504#1]
[TCP PDU reassembled in 212573]
[TCP Retransmission]
[TCP Window Full]
[TCP Dup ACK 212524#1]
Not sure if this is normal as the upload succeeds, firmware flashed and the PicoW reboots though doesn't reconnect unless I power cycle (problem for later).

Off topic question: In ota_picosdk.c the flag s_flash_irq_disabled is tested inside mg_picosdk_write() and it gets set in mg_ota_end() which means if calloc() fails in single_bank_swap() it returns before disabling interrupts so the flag will be in the wrong state for next OTA attempt? Or, am I just going mad?

Rafiki is telling me I'm overthinking all this, I'll go outside and enjoy the sunshine for a while.

[scaprile]

OK, I seem to have a clearer picture now.
Mongoose send functions do not send for TCP. Data is buffered and later sent when calling mg_mgr_poll(). One Eth frame per call.
If your code does not allow that to happen, then it won't get sent. I'm not into the Wizard code, it may have changed a lot since the last time I worked on it, and this is not the place to handle possible Wizard issues. The code I remember fired a timer before going to re-flash, in order to allow for the response to be sent.
WRT your off-topic Q, that would be my code, so I will check. It was long time ago.