LLM output arrives in bursts: a chunk of tokens lands, renders instantly, then nothing for 300ms. The result reads as stuttering rather than streaming. Pacing output to the measured input rate makes the same stream feel smooth without adding meaningful latency.
Behavior
- Measure input character rate over a sliding window, smoothed with an EWMA
- Release output at that rate, so we render about as fast as content arrives
- Never delay the first character — time-to-first-byte must not regress
- Hard cap on total added delay (mdflow uses 1s) so we can't fall behind a slow-then-fast stream
- Burst-flush the backlog when pending output exceeds a threshold
- Bypass pacing entirely when it makes no sense:
- stdin is a regular file (redirect from a file is not a live stream)
- input arrives faster than some threshold (mdflow uses 5000 chars/sec)
- stdout is not a terminal
--typewriter-off / equivalent flag
Complications
Our pipeline is synchronous: read chunk → feed() → write. Pacing needs the reader and the writer decoupled, so this likely means a reader thread plus a paced drain loop, which is the bulk of the work.
Pacing must operate on visible units, not bytes — it has to skip over SGR sequences and OSC8 hyperlinks, and treat a grapheme cluster (combining marks, emoji ZWJ sequences, regional indicators) as one tick. Otherwise emoji dribble out one codepoint at a time.
Testability: take the clock and the writer as parameters so the pacing logic can be unit-tested without sleeping.
Prior art
mdflow implements this in mdflow/typewriter.c (~700 lines, clock-injected and unit-tested). Its tokenizer, rate estimator, and pacer are cleanly separated and worth reading before we design ours.
LLM output arrives in bursts: a chunk of tokens lands, renders instantly, then nothing for 300ms. The result reads as stuttering rather than streaming. Pacing output to the measured input rate makes the same stream feel smooth without adding meaningful latency.
Behavior
--typewriter-off/ equivalent flagComplications
Our pipeline is synchronous: read chunk →
feed()→ write. Pacing needs the reader and the writer decoupled, so this likely means a reader thread plus a paced drain loop, which is the bulk of the work.Pacing must operate on visible units, not bytes — it has to skip over SGR sequences and OSC8 hyperlinks, and treat a grapheme cluster (combining marks, emoji ZWJ sequences, regional indicators) as one tick. Otherwise emoji dribble out one codepoint at a time.
Testability: take the clock and the writer as parameters so the pacing logic can be unit-tested without sleeping.
Prior art
mdflow implements this in
mdflow/typewriter.c(~700 lines, clock-injected and unit-tested). Its tokenizer, rate estimator, and pacer are cleanly separated and worth reading before we design ours.