Problems on customizing high-level HTTP Client for requesting upstream (sharing experience as well as Q&A)

[cxw620]

So far Tonic do not support proxy, so I have to write a custom HTTP client and use with EchoClient::with_origin(client, uri) (take EchoClient as an example which's generated with tonic-build).

At first, I simply use hyper::Client with my customized Connector with proxy support. However, when compression is enabled, Tonic will not work with error: protocol error: received message with compressed-flag but no grpc-encoding was specified. I've checked docs and found nothing related with such error and I have to read source codes.

• Func call tracking:
  

  tonic/tonic/src/client/grpc.rs

  Lines 280 to 367 in e31f5cc

  	pub async fn streaming<S, M1, M2, C>(
  	&mut self,
  	request: Request<S>,
  	path: PathAndQuery,
  	mut codec: C,
  	) -> Result<Response<Streaming<M2>>, Status>
  	where
  	T: GrpcService<BoxBody>,
  	T::ResponseBody: Body + Send + 'static,
  	<T::ResponseBody as Body>::Error: Into<crate::Error>,
  	S: Stream<Item = M1> + Send + 'static,
  	C: Codec<Encode = M1, Decode = M2>,
  	M1: Send + Sync + 'static,
  	M2: Send + Sync + 'static,
  	{
  	let request = request
  	.map(|s| {
  	encode_client(
  	codec.encoder(),
  	s,
  	self.config.send_compression_encodings,
  	self.config.max_encoding_message_size,
  	)
  	})
  	.map(BoxBody::new);
  	let request = self.config.prepare_request(request, path);
  	let response = self
  	.inner
  	.call(request)
  	.await
  	.map_err(Status::from_error_generic)?;
  	let decoder = codec.decoder();
  	self.create_response(decoder, response)
  	}
  	// Keeping this code in a separate function from Self::streaming lets functions that return the
  	// same output share the generated binary code
  	fn create_response<M2>(
  	&self,
  	decoder: impl Decoder<Item = M2, Error = Status> + Send + 'static,
  	response: http::Response<T::ResponseBody>,
  	) -> Result<Response<Streaming<M2>>, Status>
  	where
  	T: GrpcService<BoxBody>,
  	T::ResponseBody: Body + Send + 'static,
  	<T::ResponseBody as Body>::Error: Into<crate::Error>,
  	{
  	let encoding = CompressionEncoding::from_encoding_header(
  	response.headers(),
  	self.config.accept_compression_encodings,
  	)?;
  	let status_code = response.status();
  	let trailers_only_status = Status::from_header_map(response.headers());
  	// We do not need to check for trailers if the `grpc-status` header is present
  	// with a valid code.
  	let expect_additional_trailers = if let Some(status) = trailers_only_status {
  	if status.code() != Code::Ok {
  	return Err(status);
  	}
  	false
  	} else {
  	true
  	};
  	let response = response.map(|body| {
  	if expect_additional_trailers {
  	Streaming::new_response(
  	decoder,
  	body,
  	status_code,
  	encoding,
  	self.config.max_decoding_message_size,
  	)
  	} else {
  	Streaming::new_empty(decoder, body)
  	}
  	});
  	Ok(Response::from_http(response))
  	}
  	}

  stream ->-> create_response -> expect_additional_trailers = false -> Streaming::new_empty(decoder, body) -> Streaming::new(decoder, body, Direction::EmptyResponse, None, None)

  Here encoding is None though actually Gzip, then error occurs.

In func stream, inner T call the request and gets response from upstream. By comparing the response headers between offcial example and my own one, I noticed that grpc-status, grpc-status-message and grpc-status-details-bin were removed when in offcial example. I guessed that tonic Channel will handle gRPC Status before decoding response and remove grpc-status, grpc-status-message and grpc-status-details-bin from response headers, while hyper::Client will not do so.

Finally, I write a wrapper of hyper::Client and impl tower::Service for it, then check tonic::Status and remove grpc-status, grpc-status-message and grpc-status-details-bin from response headers. It works!

I'm not sure if I'm right and I would be really happy if anyone could share more details in the comment area.

P.S. Any convenient ways on tracing function calls in Rust?

[clouedoc]

I'm interested in using proxies with Tonic but I'm a Rust newbie and it feels like I'll end up working on it for 3 days before ending up finding weird bugs (like compression not working – thanks for mentioning it, since it's rather critical to my use case...)

Perhaps the simplest way to go about this would be to fork tonic and add the functionality there?
I'm going to pursue this approach in my fork and report here.

[clouedoc]

EDIT: I ended up patching tonic with success to support SOCKS5 proxies: #1206 (comment)

There were no issues connecting to the target service despite compression enabled, so I would guess the gRPC metadata headers were preserved. Same with TLS, I didn't need to do anything in particular despite the target service requiring mutual TLS (mTLS). In fact, I had to remove the SslConnector that I copied from your code from 2022, because it messed up with the TLS configuration I already had.

I suspect that patching Tonic is the simplest approach going forward for bugfixes or feature implementation.

A lot of thanks for your previous messages from 2022 on the other discussion; without it, it would have took me a lot of time to figure it out... It ended up taking me 2 hours instead of 3 days. Cheers :-)