Map Host header to :authority pseudo-header in HTTP/2

[sirfz]

I bumped into this issue when using pyreqwest (a Python binding for reqwest) where I hit an API (I believe it's envoy edge-proxy) which proxies the url path to the host defined in the Host header. For example GET https://proxy-host.net/hello -H 'Host: actual-host.net', the proxy server "proxies" the /hello request to actual-host.net.

When using pyreqwest with HTTP/2 enabled, the request fails (503 from the server) because the Host header is stripped out of the request (since it's not allowed in HTTP/2).

However, when using curl, the request succeeds as curl maps the Host header to :authority pseudo-header when using HTTP/2 and this seems to be the behavior of other HTTP clients (such as Python's httpx).

Is this something that can be somehow supported by reqwest?

Referencing the original issue in pyreqwest: MarkusSintonen/pyreqwest#3

[sirfz]

Curl example demonstrating expected behavior (Host being mapped to :authority):

$ curl --http2-prior-knowledge https://httpbin.org/headers -H 'Host: foo.bar'
{
  "headers": {
    "Accept": "*/*",
    "Host": "foo.bar",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-6968c785-117bc24e65eef42c3b66a566"
  }
}

Host value in the returned JSON by httpbin matches the one from the header, meaning curl managed to pass this message.

with pyreqwest:

In [1]: from pyreqwest.client import SyncClientBuilder

In [2]: c = SyncClientBuilder().base_url("https://httpbin.org/").http1(False).http2(True).build()

In [3]: r = c.get("/headers").headers({"Host": "foo.bar"}).build()

In [4]: r.send().json()
Out[4]:
{'headers': {'Accept': '*/*',
  'Accept-Encoding': 'zstd,gzip,deflate,br',
  'Host': 'httpbin.org',
  'User-Agent': 'python-pyreqwest/1.0.0',
  'X-Amzn-Trace-Id': 'Root=1-6968c6ab-1292f3d32dc66fc873c395fd'}}

Host header was dropped and never sent to the server; httpbin simply returned httpbin.org Host (same behavior if I drop the Host header from the request).

[seanmonstar]

I think I understand what's going on. You want to separate the URL-as-a-connect-target, from the URL-as-the-http-request-target? So, connect to httpbin.org, but otherwise treat the entire request as if it it were to https://foo.bar?

[sirfz]

I honestly don't know how this works. My understanding is that the :authority pseudo-header can be set to something different than the actual url authority, so the connection is established with httpbin.org but the :authority pseudo-header is set to foo.bar. Not sure if my understanding is correct or if this is even valid.

[seanmonstar]

Let me break down some of the concepts:

• In HTTP/1, the request can be GET http://httpbin.org/post HTTP/1.1\r\nHost: foo.bar\r\n\r\n. That is considered bad practice, and servers are supposed to ignore the Host header if the request-target (the part between the METHOD and the VERSION) includes a host. It's more common to have the format of GET /post HTTP/1.1\r\nHost: httpbin.org\r\n\r\n.
• In HTTP/2, a request has :authority = httpbin.org, :scheme = http, and path = /post. The Host header is illegal.

That is just the semantics of the bytes written on the socket. Most clients also conflate the idea of who to connect to by looking at the provide URL.

I suspect what you're asking about is separating those two things. Being able to say "connect to httpbin.org", and then send a request as "foo.bar".

[sirfz]

I understand, but other clients (e.g. curl/httpx) seem to support this behavior (map host to authority) so I was wondering if something can be done about this to support in reqwest/hyper as well

[goffrie]

I also ran into this today - wanting to separate which URL is used to connect to the server (and, importantly, the connection's key in the connection pool), and which URL is actually requested from the server.

I've been using roughly client.get(connect_target.join(actual_url.path())).header(HOST, actual_url.authority()) to do this on HTTP/1. This is admittedly quite hacky, but it is the most convenient way to do this. With HTTP/2 though I don't think there is any way to do it via reqwest today, so I'm just setting http1_only() on my client.

[seanmonstar]

Could configuring a Proxy on the reqwest client be the solution?

[sirfz]

Could configuring a Proxy on the reqwest client be the solution?

Just had the time to try this out and it works!

In [1]: from pyreqwest.proxy import ProxyBuilder

In [2]: from pyreqwest.client import SyncClientBuilder

In [3]: p = ProxyBuilder.all("https://httpbin.org")

In [4]: c = SyncClientBuilder().base_url("http://foo.bar").http2_prior_knowledge().proxy(p).build()

In [5]: c.get("/headers").headers({"test": "hello"}).build().send().json()
Out[5]:
{'headers': {'Accept': '*/*',
  'Accept-Encoding': 'zstd,gzip,deflate,br',
  'Host': 'foo.bar',
  'Test': 'hello',
  'User-Agent': 'python-pyreqwest/1.0.0',
  'X-Amzn-Trace-Id': 'Root=1-6980a783-7c2f7c015cc92b66216b0994'}}

[mickaelmagniez]

Is the Host > :authority header mapping something doable?

The proxy workaround will not work in my case because my application receive a lot of different couples of url/host.

Seems it's what curl is doing, documentation says

Specific http headers
Host:
The specified hostname is used for cookie matching if the cookie engine is also enabled for this transfer. If the request is done over HTTP/2 or HTTP/3, the custom hostname is instead used in the ":authority" header field and Host: is not sent at all over the wire.

Since we can't set manually the :authority header, i think the host mapping is a good solution.

Meanwhile, i'll force http1 on my client, but that's not ideal.

[sirfz]

I agree, the proxy solution is not ideal as I usually share a single pool based on base_url with many callers and using proxy forces me to create different pools for each base_url+proxy (while I should be able to maintain a single pool to the proxy regardless of the base_url being called)

[mickaelmagniez]

seems there is an ongoing PR on h2

just test locally and it fixes my issue

[patch.crates-io]
h2 = { git = "https://github.com/magurotuna/h2.git", branch = "canonicalize-authority-and-host" }