Fix retry-budget exhaustion and correlated reused-connection death (#1354)

* Fix retry-budget exhaustion and correlated reused-connection death

A client sending periodic bursts against a peer that silently discards
idle pooled connections saw transient errors escape HTTP.request with
default retry=true after ~50 burst cycles, permanently (#1353). Root
cause: every armed high-level retry consumed 10 of the per-host
RetryBucket's 500 tokens even when the retried attempt succeeded, the
bucket never refilled, and once empty the denial was silent — so the
budget was a fuse that burned out in ~50 recoveries, after which every
transient failure surfaced raw.

- RetryBucket: refund the reservation when a retried attempt reaches a
  non-retryable response (retryable 408/429/5xx responses keep partial
  cost, exception outcomes keep full cost), and credit 1 unit per
  successful non-retried request so a drained partition heals from
  healthy traffic. A depleted-partition counter keeps the per-request
  replenish check lock-free while all partitions are full.
- Transport: retry a replayable idempotent request while failures keep
  landing on *reused* pooled connections (bounded by max_idle_per_host
  + 1 acquisitions) instead of exactly once, so a correlated-death
  batch is burned through down to a fresh dial without consuming
  high-level retry budget; classify TLSError on reused connections by
  its cause so dead reused TLS connections take this path too.
- Add RetrySkippedEvent so a denied (:retry_bucket) or deadline-
  preempted (:deadline) retry is observable in request traces.
- Add TLSTransportError for TLS I/O failures on established
  connections; handshake failures are now typed TLSHandshakeError at
  the dial sites instead of one blanket wrap that mislabeled
  mid-request read errors as handshake errors. isrecoverable unwraps
  both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Cover TLS dial-site wrapping and RetrySkippedEvent emission paths

Add tests for the handshake-phase TLSHandshakeError wrap on both the
HTTP/1 transport dial and connect_h2!, the request-path (exception)
RetrySkippedEvent emission, and the verbose trace formatting of
RetrySkippedEvent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(client): harden retry and transport recovery

Make retry-budget accounting exact across built-in and custom
policies. Recover poisoned pooled connections without leaking slots or
reusing a failed connection. Normalize established TLS failures across
HTTP/1 and HTTP/2 public client boundaries.

Add regression coverage for terminal accounting, trace failures,
one-shot bodies, concurrent pool handoff, and truncated TLS records.

Fixes #1353

* Centralize the TLS truncation-message classification

Both retry classifiers matched Reseau's truncated-TLS-stream message with
a duplicated string literal; hoist it to a single documented constant so
a Reseau wording change is a one-line fix, and note that the end-to-end
truncation tests pin the coupling. Also smooth the retry-budget
CHANGELOG entry into readable prose.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: quinnj

CHANGELOG.md

@@ -10,12 +10,51 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 - Added `HTTP2Settings` to configure HTTP/2 receive flow-control windows (per-stream `initial_window_size` and connection-level `connection_window_size`). Pass it via the `http2_settings` keyword on `Client`, `Server`, `listen!`, `serve!`, `serve`, and `connect_h2!`. Defaults preserve the protocol-default 65535-byte windows, and the per-stream receive buffer cap is derived from the window. Raising the windows improves single-stream throughput on links with non-trivial latency.
 - Added `HTTP.peeraddr(::HTTP.Stream)`, returning the remote (client) `SocketAddr` of a server stream for both plain-TCP and TLS connections and both HTTP/1 and HTTP/2. This is the supported way to obtain the client IP (for rate limiting, audit logging, and per-client policy) without reaching into transport internals, and restores the capability `Sockets.getpeername(::HTTP.Stream)` provided in HTTP.jl 1.x.
+- Added `HTTP.RetrySkippedEvent`, a request trace event emitted when the retry
+  policy wanted to retry an attempt but the retry was not armed — because the
+  transport's `RetryBucket` denied capacity (`reason = :retry_bucket`) or the
+  request deadline preempted the backoff (`reason = :deadline`). Previously a
+  denied retry was indistinguishable from a non-retryable failure. ([#1353])
+- Added `HTTP.TLSTransportError`, raised when TLS-level I/O fails on an
+  established connection during a request. Previously such failures were
+  mislabeled `TLSHandshakeError`; that type is now reserved for actual
+  connection-setup failures. `HTTP.isrecoverable` classifies both wrappers by
+  their underlying cause. ([#1353])
 
 ### Fixed
 - Restored HTTP and WebSocket server task scheduling to Julia's `:interactive`
   thread pool so default-pool compute work cannot starve server and health-check
   tasks when an interactive thread is configured. ([#1342])
 - Percent-decode `userinfo` before building the `Basic` auth header (RFC 3986); fixes wrong credentials for request URLs and proxies containing percent-encoded characters.
+- Fixed the client retry budget (`RetryBucket`) treating a successful retried
+  attempt as a full-cost failure. The per-host budget drained by 10 of 500
+  units on every retry — even one that recovered with a 2xx — and never
+  refilled, so after ~50 retries against a host every subsequent retry was
+  silently denied for the transport's lifetime and transient errors surfaced
+  raw despite `retry=true`. A retry reservation is now settled by the
+  effective retry decision for the response it produced: refunded in full when
+  the built-in policy (or a custom `retry_if`) no longer wants a retry, and
+  keeping the partial cost when the response is still classified as a failure.
+  On the final attempt the built-in classification applies without invoking
+  `retry_if`, and a retry that `retry_if` explicitly requested conservatively
+  keeps cost on a non-2xx/3xx outcome. Each successful non-retried request
+  restores one unit of previously consumed budget, retry reservations and
+  response connections are released even when a trace or retry-policy callback
+  throws, and the request deadline is rechecked after the backoff sleep.
+  ([#1353])
+- The HTTP/1 transport now retries a replayable idempotent request for as long
+  as failures land on *reused* pooled connections. It tries at most
+  `max_idle_per_host` reused connections, then forces a fresh dial instead of
+  accepting another concurrent pool return. PUT and DELETE receive the same
+  stale-connection recovery as the other idempotent methods.
+  Pooled connections can be discarded by the peer in correlated batches, in
+  which case the single retry would draw the next equally-dead pooled
+  connection and fail. ([#1353])
+- Dead reused TLS connections that fail with `Reseau.TLS.TLSError` (for
+  example an RST surfacing as a wrapped `SystemError` or a truncated TLS record
+  reported as `unexpected EOF`) are now classified by their public error shape
+  in the transport's reused-connection retry. HTTP/2 read-loop wrappers also
+  preserve this classification. ([#1353])
 
 ## [v2.0.0] - 2026-04-27
 HTTP.jl 2.0 is a major rewrite of the package internals and public API. The
@@ -826,3 +865,4 @@ See changes for 0.9.15: this release is equivalent to 0.9.15 with [#752] reverte
 [#1126]: https://github.com/JuliaWeb/HTTP.jl/issues/1126
 [#1127]: https://github.com/JuliaWeb/HTTP.jl/issues/1127
 [#1342]: https://github.com/JuliaWeb/HTTP.jl/issues/1342
+[#1353]: https://github.com/JuliaWeb/HTTP.jl/issues/1353

docs/src/api/client.md

@@ -43,6 +43,7 @@ HTTP.isaborted
 HTTP.RequestEvent
 HTTP.ResponseHeadEvent
 HTTP.RetryEvent
+HTTP.RetrySkippedEvent
 HTTP.RedirectEvent
 HTTP.DoneEvent
 ```

docs/src/api/core.md

@@ -27,6 +27,7 @@ HTTP.TooManyRedirectsError
 HTTP.ConnectError
 HTTP.DNSError
 HTTP.TLSHandshakeError
+HTTP.TLSTransportError
 HTTP.AddressInUseError
 ```
 

docs/src/guides/client.md

@@ -511,6 +511,7 @@ logger — pass a `trace` callback. The callback receives subtypes of
 - [`HTTP.RequestEvent`](@ref) — request being sent
 - [`HTTP.ResponseHeadEvent`](@ref) — response headers received
 - [`HTTP.RetryEvent`](@ref) — retry scheduled
+- [`HTTP.RetrySkippedEvent`](@ref) — retry denied by the budget or deadline
 - [`HTTP.RedirectEvent`](@ref) — redirect followed
 - [`HTTP.DoneEvent`](@ref) — request finished (with response or error)
 

src/HTTP.jl

@@ -78,7 +78,8 @@ include("http_websockets.jl")
         :Handlers, :Headers, :NoProxy, :ParseError, :ProtocolError, :ProxyConfig,
         :ProxyFromEnvironment, :ProxyURL, :RedirectEvent, :Request, :RequestContext,
         :RequestEvent, :RequestRetryError, :Response, :ResponseHeadEvent, :RetryBucket,
-        :RetryEvent, :SSEEvent, :SSEStream, :Server, :StatusError, :Stream, :TLSHandshakeError,
+        :RetryEvent, :RetrySkippedEvent, :SSEEvent, :SSEStream, :Server, :StatusError, :Stream,
+        :TLSHandshakeError, :TLSTransportError,
         :TimeoutError, :TooManyRedirectsError, :Transport, :addtrailer, :appendheader,
         :body_close!, :body_closed, :body_read!, :cancel!, :canceled, :canonical_header_key,
         :close_idle_connections!, :defaultheader!, :delete, :do!, :expired, :fileserver,

src/http2_client.jl

@@ -471,8 +471,14 @@ end
 
 @inline function _throw_stream_error(conn::H2Connection, state::H2StreamState)::Nothing
     err = state.stream_error
-    err === nothing || throw(err::Exception)
-    state.conn_errored && throw(_stream_conn_error(conn))
+    if err !== nothing
+        wrapped = _wrap_tls_transport_error(err::Exception)
+        throw(wrapped)
+    end
+    if state.conn_errored
+        wrapped = _wrap_tls_transport_error(_stream_conn_error(conn))
+        throw(wrapped)
+    end
     return nothing
 end
 
@@ -1213,10 +1219,17 @@ function _connect_h2_from_tcp!(
         stream_reader = nothing
         connect_deadline_ns == 0 || TCP.set_deadline!(tcp, connect_deadline_ns)
         if secure
-            cfg = _make_tls_config_for_h2(tls_config, address)
-            tls_conn = TLS.client(tcp, cfg)
-            connect_deadline_ns == 0 || TLS.set_deadline!(tls_conn, connect_deadline_ns)
-            TLS.handshake!(tls_conn)
+            try
+                cfg = _make_tls_config_for_h2(tls_config, address)
+                tls_conn = TLS.client(tcp, cfg)
+                connect_deadline_ns == 0 || TLS.set_deadline!(tls_conn, connect_deadline_ns)
+                TLS.handshake!(tls_conn)
+            catch err
+                # TLS.client can fail while it initializes client state, before
+                # handshake! starts. Both operations are connection setup.
+                err isa TLS.TLSError && throw(TLSHandshakeError(err::TLS.TLSError))
+                rethrow()
+            end
             stream_reader = _ConnReader(tls_conn::TLS.Conn)
         else
             stream_reader = _ConnReader(tcp)
@@ -1653,7 +1666,7 @@ function body_read!(body::H2Body, dst::Vector{UInt8})::Int
             @atomic :release body.closed = true
             _clear_h2_cancel_callback!(body)
             _unregister_stream!(body.conn, body.stream_id)
-            throw(terminal_error::Exception)
+            throw(_wrap_tls_transport_error(terminal_error::Exception))
         end
         if too_many
             @atomic :release body.closed = true
@@ -1664,7 +1677,12 @@ function body_read!(body::H2Body, dst::Vector{UInt8})::Int
         end
         if nread > 0
             body.bytes_read += Int64(nread)
-            _send_window_updates!(body.conn, body.stream_id, nread)
+            try
+                _send_window_updates!(body.conn, body.stream_id, nread)
+            catch err
+                wrapped = err isa Exception ? _wrap_tls_transport_error(err::Exception) : err
+                wrapped === err ? rethrow() : throw(wrapped)
+            end
             return nread
         end
         if done
@@ -1884,5 +1902,10 @@ Send `request` over an existing `H2Connection` and return the streaming
 `Response`.
 """
 function h2_roundtrip!(conn::H2Connection, request::Request)::Response
-    return _streaming_response(_h2_roundtrip_incoming!(conn, request))
+    try
+        return _streaming_response(_h2_roundtrip_incoming!(conn, request))
+    catch err
+        wrapped = err isa Exception ? _wrap_tls_transport_error(err::Exception) : err
+        wrapped === err ? rethrow() : throw(wrapped)
+    end
 end