Zero-length StreamSend leaves QUIC_STREAM_SEND_FLAG_DATA set without a flush, permanently wedging the stream

[jbevemyr]

Version: v2.5.10, Linux (epoll datapath), built from source with OpenSSL. Reproduced deterministically.

Summary

A zero-length StreamSend (one buffer, Length = 0, no FIN/START flags) is accepted, but it can leave the stream permanently unable to transmit: every later StreamSend on that stream is accepted and queued, yet nothing ever reaches the wire, and the connection eventually dies of idle timeout.

Mechanism

1. MsQuicStreamSend accepts the zero-length request; QuicStreamSendFlush enqueues it (QueuedSendOffset does not advance) and calls QuicSendSetStreamSendFlag(.., QUIC_STREAM_SEND_FLAG_DATA, ..), which newly sets DATA and queues a flush.
2. The flush finds nothing sendable for the stream (NextSendOffset == QueuedSendOffset, so QuicStreamCanSendNow is false), writes no stream frames, and completes. The DATA flag is not cleared on this path; the normal clearing happens only when stream frames are actually written.
3. Every subsequent StreamSend: QuicStreamSendFlush calls QuicSendSetStreamSendFlag(DATA) again, but in QuicSendSetStreamSendFlag the condition (Stream->SendFlags | SendFlags) != Stream->SendFlags is false (DATA is already set) and Flags.SendDelayed is not set, so QuicSendQueueFlushForStream is skipped. No flush is ever queued for the new data.
4. If unrelated send work (an ACK-eliciting exchange, a flow control update, another stream) happens to trigger a flush that writes stream frames, the state is repaired incidentally. On a quiescent connection nothing rescues the stream: it is wedged until idle timeout.

The incidental-repair path in step 4 is why this surfaces as timing-dependent flakiness in real applications rather than a hard failure.

Instrumented trace of the failing sequence (client stream; ENQ = request enqueued in QuicStreamSendFlush, QUEUEFLUSH = QuicSendQueueFlushForStream called from QuicSendSetStreamSendFlag):

ENQ len=15 off=0  qso=15 sendflags=0x0     <- normal send
QUEUEFLUSH new=0x10 had=0x0                <- DATA set, flush queued, data flows
ENQ len=0  off=15 qso=15 sendflags=0x0     <- zero-length send
QUEUEFLUSH new=0x10 had=0x0                <- DATA set, flush queued, flush writes nothing
ENQ len=11 off=15 qso=26 sendflags=0x10    <- next send: DATA already set
                                           <- no QUEUEFLUSH line: flush skipped, 11 bytes never sent

Repro

Self-contained program below. It connects to itself over loopback with an echoing peer, verifies the stream works, issues one zero-length StreamSend, waits one second so the connection goes quiescent, then sends 11 more bytes. Those bytes never reach the peer (RESULT: WEDGED). Run with SKIP_EMPTY_SEND=1 to omit the zero-length send: then it passes.

The receive-side detour in the repro (returning QUIC_STATUS_PENDING, later StreamReceiveSetEnabled(TRUE) + StreamReceiveComplete(0), mirroring how the Erlang binding switches a stream from passive to active mode) is not part of the root cause; it just keeps the connection quiescent at the right moment. The essential trigger is a zero-length send followed by quiescence before the next send.

Build and run:

gcc -O1 -o repro repro.c -I <msquic>/src/inc -L <libdir> -lmsquic -lpthread
CERT=cert.pem KEY=key.pem ./repro               # RESULT: WEDGED
CERT=cert.pem KEY=key.pem SKIP_EMPTY_SEND=1 ./repro   # RESULT: PASS

repro.c

//
// Standalone msquic repro: zero-length StreamSend after a pended receive
// was completed with StreamReceiveComplete(0) wedges the stream send path.
//
// Sequence (client stream):
//   1. send 15 bytes; server echoes them back
//   2. client receive callback returns QUIC_STATUS_PENDING (data held)
//   3. app thread: StreamReceiveSetEnabled(TRUE); StreamReceiveComplete(0)
//   4. app thread: StreamSend with one zero-length buffer, no flags
//   5. re-indicated receive consumed normally (return SUCCESS)
//   6. app thread: send 11 more bytes
//   7. EXPECT: server receives them and echoes; FAIL if nothing within 3s
//
// Run with SKIP_EMPTY_SEND=1 to omit step 4 (control; passes).
//
#include <msquic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>

static const QUIC_API_TABLE* MsQuic;
static HQUIC Registration;
static HQUIC ServerConfig;
static HQUIC ClientConfig;
static HQUIC Listener;

static const QUIC_BUFFER Alpn = { 6, (uint8_t*)"sample" };
static uint16_t Port = 4569;

static pthread_mutex_t Lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t Cond = PTHREAD_COND_INITIALIZER;

static int ClientConnected = 0;
static int FirstRecvPended = 0;      // step 2 happened
static int ReindicatedDelivered = 0; // step 5 happened
static int SecondEchoReceived = 0;   // step 7 success
static int ServerGotSecond = 0;      // server saw the 11-byte payload
static int PendNextReceive = 1;      // client cb pends the first receive only

static void set_flag(int* flag) {
    pthread_mutex_lock(&Lock);
    *flag = 1;
    pthread_cond_broadcast(&Cond);
    pthread_mutex_unlock(&Lock);
}

static int wait_flag(int* flag, int timeout_s) {
    struct timespec ts;
    clock_gettime(CLOCK_REALTIME, &ts);
    ts.tv_sec += timeout_s;
    pthread_mutex_lock(&Lock);
    while (!*flag) {
        if (pthread_cond_timedwait(&Cond, &Lock, &ts) != 0) break;
    }
    int r = *flag;
    pthread_mutex_unlock(&Lock);
    return r;
}

//
// Server side: echo every receive back on the same stream.
//
static QUIC_STATUS QUIC_API ServerStreamCb(HQUIC Stream, void* Ctx, QUIC_STREAM_EVENT* Ev) {
    (void)Ctx;
    switch (Ev->Type) {
    case QUIC_STREAM_EVENT_RECEIVE: {
        uint64_t len = Ev->RECEIVE.TotalBufferLength;
        printf("[server] receive %llu bytes\n", (unsigned long long)len);
        if (len == 11) set_flag(&ServerGotSecond);
        if (len > 0) {
            // copy out and echo
            QUIC_BUFFER* buf = malloc(sizeof(QUIC_BUFFER) + len);
            uint8_t* data = (uint8_t*)(buf + 1);
            uint64_t off = 0;
            for (uint32_t i = 0; i < Ev->RECEIVE.BufferCount; i++) {
                memcpy(data + off, Ev->RECEIVE.Buffers[i].Buffer, Ev->RECEIVE.Buffers[i].Length);
                off += Ev->RECEIVE.Buffers[i].Length;
            }
            buf->Buffer = data;
            buf->Length = (uint32_t)len;
            MsQuic->StreamSend(Stream, buf, 1, QUIC_SEND_FLAG_NONE, buf);
        }
        return QUIC_STATUS_SUCCESS;
    }
    case QUIC_STREAM_EVENT_SEND_COMPLETE:
        free(Ev->SEND_COMPLETE.ClientContext);
        break;
    case QUIC_STREAM_EVENT_SHUTDOWN_COMPLETE:
        MsQuic->StreamClose(Stream);
        break;
    default: break;
    }
    return QUIC_STATUS_SUCCESS;
}

static QUIC_STATUS QUIC_API ServerConnCb(HQUIC Conn, void* Ctx, QUIC_CONNECTION_EVENT* Ev) {
    (void)Ctx;
    switch (Ev->Type) {
    case QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED:
        MsQuic->SetCallbackHandler(Ev->PEER_STREAM_STARTED.Stream, (void*)ServerStreamCb, NULL);
        break;
    case QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE:
        MsQuic->ConnectionClose(Conn);
        break;
    default: break;
    }
    return QUIC_STATUS_SUCCESS;
}

static QUIC_STATUS QUIC_API ListenerCb(HQUIC L, void* Ctx, QUIC_LISTENER_EVENT* Ev) {
    (void)L; (void)Ctx;
    if (Ev->Type == QUIC_LISTENER_EVENT_NEW_CONNECTION) {
        MsQuic->SetCallbackHandler(Ev->NEW_CONNECTION.Connection, (void*)ServerConnCb, NULL);
        return MsQuic->ConnectionSetConfiguration(Ev->NEW_CONNECTION.Connection, ServerConfig);
    }
    return QUIC_STATUS_SUCCESS;
}

//
// Client side.
//
static QUIC_STATUS QUIC_API ClientStreamCb(HQUIC Stream, void* Ctx, QUIC_STREAM_EVENT* Ev) {
    (void)Ctx; (void)Stream;
    switch (Ev->Type) {
    case QUIC_STREAM_EVENT_RECEIVE: {
        uint64_t len = Ev->RECEIVE.TotalBufferLength;
        int pend;
        pthread_mutex_lock(&Lock);
        pend = PendNextReceive;
        PendNextReceive = 0;
        pthread_mutex_unlock(&Lock);
        if (pend) {
            printf("[client] receive %llu bytes -> PENDING (held)\n", (unsigned long long)len);
            set_flag(&FirstRecvPended);
            return QUIC_STATUS_PENDING;
        }
        printf("[client] receive %llu bytes -> consumed\n", (unsigned long long)len);
        if (len == 15) set_flag(&ReindicatedDelivered);
        if (len == 11) set_flag(&SecondEchoReceived);
        return QUIC_STATUS_SUCCESS;
    }
    case QUIC_STREAM_EVENT_SEND_COMPLETE:
        printf("[client] send complete (canceled=%u)\n", Ev->SEND_COMPLETE.Canceled);
        break;
    default: break;
    }
    return QUIC_STATUS_SUCCESS;
}

static QUIC_STATUS QUIC_API ClientConnCb(HQUIC Conn, void* Ctx, QUIC_CONNECTION_EVENT* Ev) {
    (void)Ctx;
    switch (Ev->Type) {
    case QUIC_CONNECTION_EVENT_CONNECTED:
        set_flag(&ClientConnected);
        break;
    case QUIC_CONNECTION_EVENT_SHUTDOWN_INITIATED_BY_TRANSPORT:
        printf("[client] transport shutdown status=0x%x\n",
               (unsigned)Ev->SHUTDOWN_INITIATED_BY_TRANSPORT.Status);
        break;
    case QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE:
        MsQuic->ConnectionClose(Conn);
        break;
    default: break;
    }
    return QUIC_STATUS_SUCCESS;
}

#define CHECK(x) do { QUIC_STATUS s_ = (x); if (QUIC_FAILED(s_)) { \
    printf("FAILED %s -> 0x%x\n", #x, (unsigned)s_); exit(2); } } while (0)

int main(int argc, char** argv) {
    (void)argc; (void)argv;
    int skip_empty = getenv("SKIP_EMPTY_SEND") != NULL;

    CHECK(MsQuicOpen2(&MsQuic));
    QUIC_REGISTRATION_CONFIG reg = { "repro", QUIC_EXECUTION_PROFILE_LOW_LATENCY };
    CHECK(MsQuic->RegistrationOpen(&reg, &Registration));

    QUIC_SETTINGS settings = {0};
    settings.IdleTimeoutMs = 30000;
    settings.IsSet.IdleTimeoutMs = TRUE;
    settings.PeerBidiStreamCount = 8;
    settings.IsSet.PeerBidiStreamCount = TRUE;

    CHECK(MsQuic->ConfigurationOpen(Registration, &Alpn, 1, &settings, sizeof(settings), NULL, &ServerConfig));
    QUIC_CERTIFICATE_FILE certfile = { getenv("KEY"), getenv("CERT") };
    QUIC_CREDENTIAL_CONFIG servercred = {0};
    servercred.Type = QUIC_CREDENTIAL_TYPE_CERTIFICATE_FILE;
    servercred.CertificateFile = &certfile;
    CHECK(MsQuic->ConfigurationLoadCredential(ServerConfig, &servercred));

    CHECK(MsQuic->ConfigurationOpen(Registration, &Alpn, 1, &settings, sizeof(settings), NULL, &ClientConfig));
    QUIC_CREDENTIAL_CONFIG clientcred = {0};
    clientcred.Type = QUIC_CREDENTIAL_TYPE_NONE;
    clientcred.Flags = QUIC_CREDENTIAL_FLAG_CLIENT | QUIC_CREDENTIAL_FLAG_NO_CERTIFICATE_VALIDATION;
    CHECK(MsQuic->ConfigurationLoadCredential(ClientConfig, &clientcred));

    QUIC_ADDR addr = {0};
    QuicAddrSetFamily(&addr, QUIC_ADDRESS_FAMILY_INET);
    QuicAddrSetPort(&addr, Port);
    CHECK(MsQuic->ListenerOpen(Registration, ListenerCb, NULL, &Listener));
    CHECK(MsQuic->ListenerStart(Listener, &Alpn, 1, &addr));

    HQUIC Conn;
    CHECK(MsQuic->ConnectionOpen(Registration, ClientConnCb, NULL, &Conn));
    CHECK(MsQuic->ConnectionStart(Conn, ClientConfig, QUIC_ADDRESS_FAMILY_INET, "127.0.0.1", Port));
    if (!wait_flag(&ClientConnected, 5)) { printf("FAIL: no connect\n"); return 2; }

    HQUIC Stream;
    CHECK(MsQuic->StreamOpen(Conn, QUIC_STREAM_OPEN_FLAG_NONE, ClientStreamCb, NULL, &Stream));
    CHECK(MsQuic->StreamStart(Stream, QUIC_STREAM_START_FLAG_NONE));

    // step 1
    static QUIC_BUFFER b1;
    b1.Buffer = (uint8_t*)"ping_passiveeee";
    b1.Length = 15;
    CHECK(MsQuic->StreamSend(Stream, &b1, 1, QUIC_SEND_FLAG_NONE, NULL));

    // step 2: echo arrives, client cb pends it
    if (!wait_flag(&FirstRecvPended, 5)) { printf("FAIL: first echo never pended\n"); return 2; }

    // step 3: what quicer's setopt(active, true) does
    printf("[app] StreamReceiveSetEnabled(TRUE) + StreamReceiveComplete(0)\n");
    CHECK(MsQuic->StreamReceiveSetEnabled(Stream, TRUE));
    MsQuic->StreamReceiveComplete(Stream, 0);

    // step 4: the poison
    if (!skip_empty) {
        static QUIC_BUFFER b0;
        b0.Buffer = (uint8_t*)"";
        b0.Length = 0;
        printf("[app] zero-length StreamSend\n");
        CHECK(MsQuic->StreamSend(Stream, &b0, 1, QUIC_SEND_FLAG_NONE, NULL));
    } else {
        printf("[app] (skipping zero-length StreamSend)\n");
    }

    // step 5: re-indication of the 15 bytes gets consumed
    if (!wait_flag(&ReindicatedDelivered, 5)) { printf("FAIL: pended data never re-indicated\n"); return 2; }

    sleep(1); // quiesce
    // step 6
    static QUIC_BUFFER b2;
    b2.Buffer = (uint8_t*)"ping_active";
    b2.Length = 11;
    printf("[app] sending 11 bytes\n");
    CHECK(MsQuic->StreamSend(Stream, &b2, 1, QUIC_SEND_FLAG_NONE, NULL));

    // step 7
    int server_got = wait_flag(&ServerGotSecond, 3);
    int echoed = wait_flag(&SecondEchoReceived, 3);
    printf("server received second payload: %s\n", server_got ? "yes" : "NO");
    printf("client received second echo:    %s\n", echoed ? "yes" : "NO");
    if (server_got && echoed) {
        printf("RESULT: PASS\n");
        return 0;
    }
    printf("RESULT: WEDGED\n");
    return 1;
}

Possible fixes

Either reject zero-length sends without FIN/START in MsQuicStreamSend (QUIC_STATUS_INVALID_PARAMETER), or avoid setting QUIC_STREAM_SEND_FLAG_DATA for requests with TotalLength == 0 in QuicStreamSendFlush (completing them immediately), or clear DATA when a flush finds the stream has no pending data. Any of the three breaks the wedge.

Context

Found while debugging a flaky test in the Erlang binding quicer (emqx/quic#441): an application-level empty send wedged streams in a way that looked like receive-path reordering until traced down to this send-scheduler state.

[Tani-shar]

Reproduces on main (4984c21, 2.7.0), Ubuntu, OpenSSL backend, x64.

RESULT: WEDGED. With SKIP_EMPTY_SEND=1: RESULT: PASS. Behavior unchanged since 2.5.10.

I'd like to look at the fix — is anyone already on this?

[guhetier]

Tanishq Sharma (@Tani-shar) Please feel free to open a PR.
The team is pretty busy right now and it might take us a bit to either review a PR or address the issue ourselves.

This area is generally (both in terms of synchronization and state machine). The fix is likely one or two simple "if" tests to avoid queueing a send operation that result in a no-op + to avoid getting stuck if ever processing one.

[Tani-shar]

Opened #6248 with a fix. I went with your second suggestion — skipping the DATA flag when TotalLength == 0 — rather than clearing DATA on a failed flush, since QuicStreamSendCanWriteDataFrames returns false for flow-control blocking and peer-not-allowed too, so that version would wedge differently.

Your repro reproduces on main at 4984c21 and passes with the fix. Thanks for the writeup — the trace made this straightforward to follow.

Guillaume Hetier (@guhetier) You mentioned a second guard to avoid getting stuck if a no-op send is ever processed — I only did the first (don't queue it). Happy to add the second if you'd like it in the same PR.

[Tani-shar]

Update: added the second guard and a regression test.

Guard 2 clears QUIC_STREAM_SEND_FLAG_DATA in the stream write path when the DATA flag is set but QuicStreamHasPendingStreamData returns false. That function compares send offsets only, so it doesn't fire when data is pending but blocked by flow control or by the peer not yet allowing the stream.

Misc.StreamZeroLengthSend fails on main at 4984c21 and passes with the fix. Each guard breaks the wedge independently, I verified by disabling one at a time.