Skip to content

Performance: Set PSH only on the segment that empties the send queue - #1381

Open
comotion wants to merge 3 commits into
FreeRTOS:mainfrom
comotion:tcp-psh-last-segment
Open

comotion wants to merge 3 commits into
FreeRTOS:mainfrom
comotion:tcp-psh-last-segment

Conversation

@comotion

@comotion comotion commented Nov 3, 2026

Copy link
Copy Markdown

Set PSH only on the segment that empties the send queue

Description

A Linux peer ACKs every segment that carries PSH at once, so setting PSH on every data segment makes a sender with several segments queued get one ACK per segment. The stack has to process each ACK in order, which takes CPU time from sending, reducing throughput.

  • ipconfigTCP_PSH_ON_LAST_SEGMENT_ONLY sets PSH only on the segment that carries the last queued byte, as other stacks do.
  • It is off by default, and nothing changes while it is off.

Test Steps

Measure transmit speeds for MSS, 16KB and 256KB transmits.
My measurements are on a 125MHz Nios-V with a msgDMA ethernet driver and a 10GbE PHY.
On TX, 16KB sends, this is 27% faster than baseline, and on 256KB sends, 28% faster.
Single-MSS sends are unchanged.

Checklist:

  • I have tested my changes. No regression in existing tests.
  • I have modified and/or added unit-tests to cover the code changes in this Pull Request.

Related Issue

#1380

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

A Linux peer ACKs every segment that carries PSH at once, so setting
PSH on every data segment makes a sender with several segments queued
get one ACK per segment.

- ipconfigTCP_PSH_ON_LAST_SEGMENT_ONLY sets PSH only on the segment
  that carries the last queued byte, as other stacks do.
- It is off by default, and nothing changes while it is off.

@htibosch htibosch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@comotion , thank you for sharing this change.
You are right, the throughput increases dramatically when tcpTCP_FLAG_PSH is set when the last segment has been sent. It saves ACKs: less bandwidth needed.

We chose to set PSH in every packet that contains TX data.

I just ran iperf3, and tried it as: iperf3 -c 192.168.1.3 -4 --port 5001 --bytes 100M -R.

-R makes that the DUT will send packets to the client. The performance looked good.

Changing this behaviour means that the counterpart must be able to accommodate incoming packets that have not been consumed.
Although, the counterpart can also just ignore the flag and consume the data.

I just see a WireShark option, when you view a TCP packet:

[SEQ/ACK analysis]
    [iRTT: 1.120900 milliseconds]
    [Bytes in flight: 1460]
    [Bytes sent since last PSH flag: 5840]

Should we limit what we send "Bytes sent since last PSH flag" by counting them and insert a PSH flag?

I want to play more with the change and see how it behaves before approving it.

Thanks,
Hein

@comotion

comotion commented Nov 9, 2026

Copy link
Copy Markdown
Author

Hi @htibosch and thanks for taking the time to review and test.

You have a point that if an application keeps the send buffer busy, this change never sets a PSH and some receivers could hold the data until a PSH arrives. Linux reads the data straight away, Windows might wait for a PSH in some cases, I haven't tested.

We can guard against this in the same way Linux does in forced_push(), by sending a PSH once more than half the peer's advertised window has gone out since the last PSH. Adding such a check won't affect performance.
Since we can never test against all receivers out there, adding such a change would guard against that case.

A sender that keeps its queue full never empties it, so PSH would never be
set. A receiver that waits for PSH before it hands data to the application
then holds the data until its buffer fills. Linux sets PSH in the same case.
The proof now also runs with PSH set only on the last segment, so CI checks the code that the option compiles in.
@htibosch

Copy link
Copy Markdown
Contributor

@comotion I like how you used of the circular buffers: roughly it says:

    if( this is likely the last TX packet )
    {
        Set the PSH flag
    }

For my self and others, a short comment on TCP sliding windows and circular buffers.

When implementing TCP with sliding windows, we didn't want to use pbuf, in stead each Ethernet packet is stored in a contiguous block, not in linked chunks.

Each TCP socket may allocate two circular buffers, one for RX and one for TX. Their sizes are configured:

#define ipconfigTCP_RX_BUFFER_LENGTH    ( 4 * ipconfigTCP_MSS )
#define ipconfigTCP_TX_BUFFER_LENGTH    ( 4 * ipconfigTCP_MSS )

Note that you can configure all properties with the socket option FREERTOS_SO_WIN_PROPERTIES.

Back to the circular buffers:

Bytes are added to the position "head"
Bytes are taken from the position "tail"

As soon as "head == tail", the buffer is empty. The markers tail and head are always moving forward.

Note this function:

    size_t uxStreamBufferGetSize( const StreamBuffer_t * const pxBuffer )
    {
        return uxStreamBufferDistance( pxBuffer, pxBuffer->uxTail, pxBuffer->uxHead );
    }

When the distance is 0, the buffer is empty
Otherwise, some math is needed:

    const size_t uxLength = pxBuffer->LENGTH;
    size_t uxCount = uxLength + uxHead - uxTail;

    if( uxCount >= uxLength )
    {
        uxCount -= uxLength;
    }

    return uxCount;

The PSH will be sent when this transfer is sending the last bytes, up until txStream->uxHead.

   lDataLen = ( int32_t ) ulTCPWindowTxGet( pxTCPWindow, pxSocket->u.xTCP.ulWindowSize, &lStreamPos );

lDataLen : The number of bytes that can be sent
lStreamPos : The index within the TX stream buffer of the first byte to be sent.

    if( lDataLen != 0L )
    {
        size_t uxDistance = uxStreamBufferDistance( txStream, ( size_t ) lStreamPos, txStream->uxHead );
        if( uxDistance == ( size_t ) lDataLen )
        {
            pxProtocolHeaders->xTCPHeader.ucTCPFlags |= ( uint8_t ) tcpTCP_FLAG_PSH;
        }
    }

Here an overview of the cyclic buffers and markers:

plus_tcp_sliding_windows

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants