A Stop-and-Wait ARQ (Automatic Repeat reQuest) reliable communication library built on top of raw UDP sockets in Go. The library guarantees message delivery via a timeout-retry mechanism with acknowledgments - the sender blocks until an ACK is received for every sent message.
Usage: sender -addr <receiver_ip:port> -n <num_messages> -size <payload_size>
Flags:
-addr string receiver address (e.g., "10.7.9.7:9002")
-n int number of messages to send (default: 1000)
-size int payload size in bytes (default: 100)
Behavior:
- Sends N messages to the receiver
- Measures and prints RTT for each message
- Prints summary: min/avg/max/p99 RTT, total time, throughput
Usage: receiver -port <listen_port> -drop <drop_percent>
Flags:
-port int UDP port to listen on (default: 9002)
-drop float drop percentage 0-100 (default: 0)
Behavior:
- Listens for DATA packets
- Drops packets at the configured rate (simulated loss)
- Sends ACKs for non-dropped packets
- Prints stats: packets received, dropped, total
Usage: bandwidth -mode <sender|receiver> -addr <ip:port> -n <num_packets>
Behavior:
- Sends N max-sized packets (1423-byte payload) back-to-back
- Measures total time, throughput in MB/s and Mbps
- Reports overhead per packet
part2/
├── bin/ # compiled executables for optimized and non-optimized builds
├── go.mod
├── Makefile # build scripts for optimized and non-optimized binaries
├── reliableudp/
│ ├── conn.go # Conn struct, NewConn, Send, Listen, Close
│ └── packet.go # Packet struct, Marshal, Unmarshal, constants
├── cmd/
│ ├── sender/
│ │ └── main.go # CLI sender: sends N messages, records RTTs
│ ├── receiver/
│ │ └── main.go # CLI receiver: listens, applies drop rate
│ └── bandwidth/
│ └── main.go # Bandwidth benchmark: max-size packet flood
│
├── raw_results/
│ ├── media/
│ ├── localnet/
│ └── loopback/
│
├── performance_reliability_measurements.csv
├── optimization_analysis.csv
├── reliability_analysis/
│ ├── localnet_no_opt_rtt_drop_0.png
│ ├── localnet_no_opt_rtt_drop_5.png
│ ├── ...
│ ├── localnet_opt_rtt_drop_50.png
│ ├── .....
│ ├── loopback_no_opt_rtt_drop_0.png
│ ├── loopback_no_opt_rtt_drop_5.png
│ ├── ...
│ └── loopback_opt_rtt_drop_50.png
└── README.md
# builds executables in bin/ for both optimized and non-optimized versions
make all
# --- Same machine ---
# Terminal 1: start receiver
./bin/receiver-opt -port 9002 -drop 0
# Terminal 2: send messages
./bin/sender-opt -addr "127.0.0.1:9002" -n 10000 -size 100
# --- Cross machine ---
# On Machine B (10.7.X.X):
./bin/receiver-opt -port 9002 -drop 10
# On Machine A (10.7.7.68):
./bin/sender-opt -addr "10.7.X.X:9002" -n 1000 -size 100
# --- Bandwidth test ---
# Machine B:
./bin/bandwidth-opt -mode receiver -addr ":9002"
# Machine A:
./bin/bandwidth-opt -mode sender -addr "10.7.X.X:9002" -n 10000All experiments send 1,000 messages of 100 bytes each (RTT/reliability) or 10,000 max-sized packets of 1,427 bytes (bandwidth).
Machines: loopback 127.0.0.1 (same host) and cross-machine 10.7.9.7 (friend's machine on college LAN, 10.7.0.0/18).
This is the baseline latency with a perfectly reliable channel.
| Scenario | Build | Avg RTT | Min RTT | P50 RTT | P99 RTT | Max RTT | Stddev | Throughput |
|---|---|---|---|---|---|---|---|---|
| Loopback | Optimized | 52.7 µs | 23.1 µs | 42.0 µs | 135.8 µs | 290.3 µs | 29.1 µs | 15.18 Mbps |
| Loopback | Non-optimized | 45.5 µs | 21.8 µs | 34.5 µs | 109.4 µs | 237.0 µs | 24.9 µs | 17.58 Mbps |
| Local Network | Optimized | 7.75 ms | 5.79 ms | 7.52 ms | 11.74 ms | 84.1 ms | 2.59 ms | 0.10 Mbps |
| Local Network | Non-optimized | 7.94 ms | 5.59 ms | 7.53 ms | 12.42 ms | 132.0 ms | 4.62 ms | 0.10 Mbps |
Key observations:
- Loopback RTT (~50 µs) is dominated by two
recvfromsyscalls and the ACK encode/decode path - there is no real network traversal. - Cross-machine RTT (~7.7 ms) is ~150x higher than loopback, reflecting actual kernel → NIC → switch → NIC → kernel latency on the college LAN. The P50 of 7.5 ms is very stable; the long P99/max tail is caused by occasional OS scheduling jitter on the peer machine.
- The non-optimized build is marginally faster on loopback because disabling inlining changes the function call boundary layout in a way that happens to reduce branch mispredictions for the very tight encode/decode loop. On cross-machine runs where network latency dominates, the difference disappears.
Varying the receiver-side drop rate reveals the three performance regimes of Stop-and-Wait ARQ.
| Drop % | Retries | Avg RTT | P50 RTT | P99 RTT | Max RTT | Total Time |
|---|---|---|---|---|---|---|
| 0% | 0 | 52.7 µs | 42.0 µs | 135.8 µs | 290.3 µs | 52.7 ms |
| 5% | 60 | 12.2 ms | 142.8 µs | 201.3 ms | 402.6 ms | 12.2 s |
| 10% | 98 | 19.8 ms | 141.7 µs | 202.4 ms | 603.7 ms | 19.8 s |
| 25% | 382 | 76.9 ms | 159.6 µs | 603.6 ms | 805.0 ms | 1 m 17 s |
| 50% | 1,094 | 220.2 ms | 200.6 ms | 1,407.8 ms | 2,212.9 ms | 3 m 40 s |
Performance regimes visible in the RTT-vs-message-number graphs:
- Normal delivery (P50 ~ 50 µs) - the vast majority of messages are delivered on the first attempt. The RTT is just the network round-trip plus encode/decode overhead.
- Single retransmit (RTT ~ timeout + RTT ~ 200 ms + 50 µs) - the timeout fires once, the packet is resent, and the ACK arrives. The graph shows periodic spikes at exactly ~200 ms intervals.
- Multiple retransmits (RTT ~ N x timeout) - at 50% drop, many messages require 2–4 retransmits. The P99 of 1.4 s represents ~7 consecutive timeouts for the same message. Total throughput collapses to effectively zero.
The P50 RTT stays near the no-drop value even at high drop rates because most messages still get through on the first try - retransmission events are visible as outlier spikes in the RTT plot.
| Drop % | Retries | Avg RTT | P50 RTT | P99 RTT | Max RTT |
|---|---|---|---|---|---|
| 0% | 0 | 7.75 ms | 7.52 ms | 11.74 ms | 84.1 ms |
| 5% | 57 | 19.4 ms | 7.53 ms | 210.8 ms | 410.2 ms |
| 10% | 126 | 33.6 ms | 7.56 ms | 410.7 ms | 412.3 ms |
| 25% | 338 | 77.0 ms | 7.82 ms | 612.5 ms | 1,629.7 ms |
| 50% | 968 | 204.6 ms | 13.7 ms | 1,216.6 ms | 2,023.1 ms |
Behaviour mirrors loopback but with a 150x higher base RTT. At 50% drop, the P50 shifts to 13.7 ms (from 7.5 ms) because even "successful" deliveries increasingly require one prior retry - the probability that at least one attempt in a row succeeds is low enough that many messages need 1–2 retries before the P50.
10,000 packets x 1,427-byte payload = 14.27 MB total data per run.
| Scenario | Build | Throughput (MB/s) | Throughput (Mbps) | Per-packet avg | Retries |
|---|---|---|---|---|---|
| Loopback | Optimized | 64.9 MB/s | 519.2 Mbps | 22.0 µs | 0 |
| Loopback | Non-optimized | 52.7 MB/s | 421.2 Mbps | 27.1 µs | 0 |
| Local Network | Optimized | 0.18 MB/s | 1.44 Mbps | 7.95 ms | 1 |
| Local Network | Non-optimized | 0.18 MB/s | 1.42 Mbps | 8.05 ms | 2 |
What limits the bandwidth?
- Loopback: Bottleneck is CPU throughput (encode + two syscalls per packet). With ~22 µs per packet and 1,427 bytes, the theoretical ceiling of pure Stop-and-Wait is
1427 / 22µs ~ 64 MB/s- which matches the measured value near-exactly. Disabling compiler optimizations degrades encoding speed by ~23%, directly corresponding to the throughput drop. - Local Network: Bottleneck is Stop-and-Wait latency, not the link speed. With ~8 ms RTT and 1,427-byte packets:
1427 / 8ms ~ 0.18 MB/s- again matching perfectly. The 1 Gbps link is used at only 1.44 / 1000 = 0.14% of capacity. A sliding-window protocol with window size W can recoverW x 0.18 MB/sof the wasted bandwidth.
| Metric | Scenario | Optimized | Non-optimized | % speedup |
|---|---|---|---|---|
| Avg RTT (drop=0) | Loopback | 52.7 µs | 45.5 µs | −14% (no-opt faster) |
| P50 RTT (drop=0) | Loopback | 42.0 µs | 34.5 µs | −21% (no-opt faster) |
| P99 RTT (drop=0) | Loopback | 135.8 µs | 109.4 µs | −24% (no-opt faster) |
| RTT throughput (drop=0) | Loopback | 15.18 Mbps | 17.58 Mbps | −14% (no-opt faster) |
| Avg RTT (drop=0) | Local Network | 7.75 ms | 7.94 ms | +2.5% (opt faster) |
| P99 RTT (drop=0) | Local Network | 11.74 ms | 12.42 ms | +5.5% (opt faster) |
| Bandwidth | Loopback | 519.2 Mbps | 421.2 Mbps | +23.3% |
| Bandwidth | Local Network | 1.44 Mbps | 1.42 Mbps | +1.4% |
Interpretation:
- On loopback, the non-optimized binary is slightly faster for RTT. This counter-intuitive result occurs because with
-gcflags="all=-N -l"the functions are not inlined, which subtly changes the memory layout of hot variables in the encode/decode loop - in this case landing them in a register-friendly layout that reduces latency for the 5-byte header read. The effect is small (< 25 µs) and would disappear with larger payloads. - For bandwidth on loopback, the optimized binary wins decisively (+23%). Bandwidth is CPU-bound (encoding 10,000 packets back-to-back), so the inlining and loop unrolling that
-gcflagsdisables directly reduces throughput. - On the local network, optimization has almost no measurable effect on either metric because both are dominated by the 7–8 ms network round-trip - a 20 µs difference in encode time is invisible at that scale.
Figure 1: Setup of how RTT runs were setup - server running in one terminal, client running in another terminal on the same machine (loopback)
Figure 2: Setup of bandwidth runs - server running in one terminal, client running in another terminal on the same machine (loopback)
Figure 3: Setup of local network runs - client running on my machine, server running on friend's machine on the same college LAN.
Figure 4: Setup of how logs were recorded from the local network runs
┌─────────────────────────────────────────────────────────────────────┐
│ Application Layer │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Sender │ │ Receiver │ │
│ │ Application │ │ Application │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ Send(addr, data) │ Listen() │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Reliable UDP Library │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌───────────────┐ │ │
│ │ │ Packet │ │ Timeout │ │ Drop │ │ │
│ │ │ Codec │ │ Retry │ │ Simulator │ │ │
│ │ │ │ │ Engine │ │ (receiver) │ │ │
│ │ └────────────┘ └────────────┘ └───────────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────┐ │ │
│ │ │ Sequence Number Manager │ │ │
│ │ └──────────────────────────────────────────────┘ │ │
│ └──────────────────────┬──────────────────────────────┘ │
│ │ │
└──────────────────────────┼──────────────────────────────────────────┘
│
┌──────▼───────┐
│ net.UDPConn │
│ (Go stdlib) │
└──────┬───────┘
│
┌──────▼──────┐
│ Raw UDP │
│ Socket │
└─────────────┘
┌───────────────────────────────────┐
│ Public API │ Send() / Listen() / Close()
├───────────────────────────────────┤
│ Reliability Layer │ SeqNum tracking, ACK matching
│ (timeout/retry engine) │ Duplicate detection
├───────────────────────────────────┤
│ Codec Layer │ Binary marshal/unmarshal
│ (packet.go) │ Header + payload encoding
├───────────────────────────────────┤
│ Drop Simulation Layer │ Controlled random packet drops
│ (receiver side only) │ Configurable drop rate [0.0, 1.0)
├───────────────────────────────────┤
│ net.UDPConn (Go stdlib) │ Raw UDP send/receive
└───────────────────────────────────┘
┌──────────────────────────────────────────────────┐
│ Machine: 10.7.7.68 │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Sender │ <---> │ Receiver │ │
│ │ 127.0.0.1 │ │ 127.0.0.1 │ │
│ │ :9001 │ │ :9002 │ │
│ └──────────────┘ └──────────────┘ │
│ │ │ │
│ └───── lo0 (loopback) ───┘ │
│ │
│ Expected RTT: < 100 µs │
│ Bandwidth: limited by CPU, not network │
└──────────────────────────────────────────────────┘
┌────────────────────────┐ ┌────────────────────────┐
│ Machine A │ Subnet │ Machine B │
│ 10.7.7.68 │ 10.7.0.0/18 │ 10.7.X.X │
│ │ │ │
│ ┌──────────────┐ │ │ ┌──────────────┐ │
│ │ Sender │─────┼──── UDP ─────┼────▸│ Receiver │ │
│ │ :9001 │ │ │ │ :9002 │ │
│ └──────────────┘ │ │ └──────┬───────┘ │
│ │ │ │ │
│ ┌──────────────┐ │ │ ┌──────▼───────┐ │
│ │ (receives │◂────┼──── UDP ─────┼─────│ (sends ACK) │ │
│ │ ACK back) │ │ │ │ │ │
│ └──────────────┘ │ │ └──────────────┘ │
│ │ │ │
│ en0: 10.7.7.68 │ switch / │ en0: 10.7.X.X │
│ │ router │ │
└────────────────────────┘ └────────────────────────┘
Expected RTT: 0.5 - 5 ms (depending on switch hops)
Bandwidth: limited by link speed (likely 100 Mbps or 1 Gbps)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number (32-bit) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type (8) | |
+-+-+-+-+-+-+-+-+ +
| Payload (0 - 1427 bytes) |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Header: 5 bytes total
- SeqNum: 4 bytes (uint32, big-endian)
- Type: 1 byte (0x01 = DATA, 0x02 = ACK)
No Length field - UDP preserves datagram boundaries,
so payload size = bytes_received - 5.
Max payload: 1427 bytes (1432 total - 5 header = 1427)
Chosen to stay well under 1500-byte MTU
(1500 - 20 IP - 8 UDP - 5 header = 1467 max, we use 1427 for safety)
| Type | Value | Direction | Contains Payload |
|---|---|---|---|
DATA |
0x01 |
Sender → Receiver | Yes |
ACK |
0x02 |
Receiver → Sender | No (length = 0) |
┌──────────┐
Send() │ IDLE │
called ────▸│ │
└────┬─────┘
│
encode packet
seqNum = next
│
┌─────▼────┐
┌──────│ WAIT_ACK │◂──────┐
│ └────┬─────┘ │
│ │ │
timeout recv pkt recv pkt
expired (wrong seq) (correct ACK)
│ │ │
│ ┌────▼─────┐ ┌────▼─────┐
│ │ DISCARD │ │ DONE │
│ │ (ignore)│ │ seqNum++│
│ └──────────┘ │ return │
│ └──────────┘
│
┌────▼───────┐
│ RETRANSMIT │
│ same pkt │──────▸ back to WAIT_ACK
└────────────┘
┌──────────┐
Listen() │ IDLE │
called ────▸│ │
└────┬─────┘
│
recv packet
│
┌────▼─────┐
│ CHECK │
│ drop? │
└────┬─────┘
╱ ╲
rand < rate rand >= rate
(DROP) (ACCEPT)
│ │
┌────▼──────┐ ┌────▼──────┐
│ SILENTLY │ │ DECODE │
│ DISCARD │ │ packet │
└───────────┘ └────┬──────┘
│
┌─────▼──────┐
│ SEND ACK │
│ seq=pkt.seq│
└─────┬──────┘
│
┌─────▼──────┐
│ DELIVER │
│ to app │
└────────────┘
Sender Receiver
│ │
│━━━ DATA(seq=1, payload) ━━━━━━━━━━━▸ │ t=0ms
│ [start timer: 200ms] │
│ │── process & ACK
│◂━━━━━━━━━━━━━━━━ ACK(seq=1) ━━━━━━━ │ t=1ms
│ [timer cancelled] │
│ │
│━━━ DATA(seq=2, payload) ━━━━━━━━━━━▸ │ t=2ms
│ [start timer: 200ms] │
│ x (packet lost) │
│ [timeout @ 200ms] │
│ │
│━━━ DATA(seq=2, payload) ━━━━━━━━━━━▸ │ t=202ms (RETRANSMIT)
│ [start timer: 200ms] │
│ │── process & ACK
│◂━━━━━━━━━━━━━━━━ ACK(seq=2) ━━━━━━━ │ t=203ms
│ [timer cancelled] │
│ │
│━━━ DATA(seq=3, payload) ━━━━━━━━━━━▸ │ t=204ms
│ [start timer: 200ms] │
│ │── ACK sent but lost ╳
│ [timeout @ 200ms] │
│ │
│━━━ DATA(seq=3, payload) ━━━━━━━━━━━▸ │ t=404ms (RETRANSMIT)
│ [start timer: 200ms] │
│ │── duplicate detected, re-ACK
│◂━━━━━━━━━━━━━━━━ ACK(seq=3) ━━━━━━━ │ t=405ms
│ [timer cancelled] │
Parameters:
| Parameter | Default | Description |
|---|---|---|
Timeout |
200 ms | Time to wait for ACK before retransmitting |
MaxRetries |
inf | Sender retries indefinitely until ACK received |
DropRate |
0.0 | Fraction of packets receiver drops [0.0, 1.0) |
package reliableudp
import (
"net"
"time"
)
// Config holds tunable parameters for the reliable UDP connection.
type Config struct {
Timeout time.Duration // ACK wait timeout (default: 200ms)
DropRate float64 // receiver-side drop probability [0.0, 1.0)
}
// Conn wraps a UDP socket with reliable delivery guarantees.
type Conn struct {
udp *net.UDPConn
config Config
seqNum uint32 // next sequence number to use (sender)
lastSeen uint32 // last delivered seqNum (receiver, for dedup)
}// NewConn creates a reliable UDP connection bound to the given local address.
// Pass dropRate > 0 to simulate receiver-side packet loss.
func NewConn(listenAddr string, cfg Config) (*Conn, error)
// Send reliably delivers data to the remote address.
// Blocks until an ACK with matching sequence number is received.
// Returns the round-trip time of the successful delivery.
func (c *Conn) Send(remoteAddr *net.UDPAddr, data []byte) (rtt time.Duration, err error)
// Listen blocks until a DATA packet is received (after drop simulation).
// Automatically sends an ACK back to the sender.
// Returns the payload and the sender's address.
func (c *Conn) Listen() (data []byte, sender *net.UDPAddr, err error)
// Close releases the underlying UDP socket.
func (c *Conn) Close() error
// SetDropRate changes the receiver-side drop probability at runtime.
func (c *Conn) SetDropRate(rate float64)package reliableudp
const (
TypeDATA byte = 0x01
TypeACK byte = 0x02
HeaderSize = 5 // 4 (seq) + 1 (type) - no length field
MaxPayload = 1427 // fits within MTU
MaxPacket = HeaderSize + MaxPayload // 1432 bytes
)
// Packet is the wire-format structure.
type Packet struct {
SeqNum uint32
Type byte
Payload []byte
}
// Marshal encodes a Packet into wire bytes (big-endian).
func (p *Packet) Marshal() []byte
// Unmarshal decodes wire bytes into a Packet.
func Unmarshal(data []byte) (*Packet, error)Experiment: Send small messages (1 byte payload), measure total RTT.
Overhead = RTT - (expected raw network latency)
Same machine: RTT_raw ~ 0 → overhead ~ full RTT
Cross machine: RTT_raw ~ ping time → overhead = measured RTT - ping RTT
┌──────────────────────────────────────────────────────────┐
│ Experiment: Send 10,000 small messages (100 bytes) │
│ │
│ Scenarios: │
│ A) Same machine (loopback) - sender & receiver on │
│ 127.0.0.1, different ports │
│ B) Cross machine - sender on 10.7.7.68, │
│ receiver on 10.7.X.X │
│ │
│ Metrics: │
│ - Per-message RTT (record each) │
│ - Min, Avg, Max, P50, P99 RTT │
│ - Jitter (stddev of RTT) │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Experiment: Send 10,000 max-sized packets (1423 bytes) │
│ │
│ Scenarios: │
│ A) Same machine (loopback) │
│ B) Cross machine (10.7/18 subnet) │
│ │
│ Metrics: │
│ - Total bytes transferred │
│ - Total time elapsed │
│ - Throughput: MB/s and Mbps │
│ - Per-packet overhead │
│ │
│ Analysis: │
│ - What limits bandwidth? (CPU encoding, ACK wait, │
│ network link speed, syscall overhead) │
│ - Improvement ideas: pipelining, sliding window, │
│ batch ACKs, larger packets │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Experiment: Send 1,000 messages with varying drop rates │
│ │
│ Drop rates: 0%, 5%, 10%, 25%, 50% │
│ │
│ For each drop rate: │
│ - Record RTT of each message │
│ - Plot RTT vs message number (scatter plot) │
│ - Identify retransmission spikes in the graph │
│ - Count total retransmissions │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Compile and run same benchmarks: │
│ │
│ Without optimization (debug): │
│ go build -gcflags="-N -l" -o sender_debug ./cmd/sender│
│ │
│ With optimization (default): │
│ go build -o sender_opt ./cmd/sender │
│ │
│ Compare: │
│ - RTT (same machine, cross machine) │
│ - Bandwidth (same machine, cross machine) │
│ - Per-packet encoding/decoding time │
│ │
│ -gcflags explanation: │
│ -N disable optimizations │
│ -l disable inlining │
└──────────────────────────────────────────────────────────┘